2024-03-25T18:22:31.519 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:22:31.519 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/orig.config.yaml 2024-03-25T18:22:31.559 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:22:31.648 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621325' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9CW1XN++EKPZEb9HmxbLYSDkky3NYmzMESg0BgqpifYbRz3QKNUxTHlGiSxUPIXQ8WKSkuXfLz3ENkao/ZWKU= smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD61+DYpDp+6uYpwZwXyqjynu8sPDA81HZl0StzJkQ3mELz3Zi7fNYKjpR9ksHYe8nHoeAss37muvbvByDQjCbg= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:22:31.648 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:22:31.648 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:22:31.649 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:22:31.650 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:22:31.671 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:22:31.671 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:22:31.672 INFO:teuthology.packaging:ref: None 2024-03-25T18:22:31.672 INFO:teuthology.packaging:tag: None 2024-03-25T18:22:31.672 INFO:teuthology.packaging:branch: main 2024-03-25T18:22:31.672 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:22:31.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:22:31.900 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:22:31.901 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:22:31.908 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:22:31.908 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:22:31.915 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:22:31.929 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:22:31.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:22:31.960 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.651068', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9CW1XN++EKPZEb9HmxbLYSDkky3NYmzMESg0BgqpifYbRz3QKNUxTHlGiSxUPIXQ8WKSkuXfLz3ENkao/ZWKU='} 2024-03-25T18:22:31.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.649769', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD61+DYpDp+6uYpwZwXyqjynu8sPDA81HZl0StzJkQ3mELz3Zi7fNYKjpR9ksHYe8nHoeAss37muvbvByDQjCbg='} 2024-03-25T18:22:31.984 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:22:31.990 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:22:31.990 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:22:31.990 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:22:32.069 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f43cc1d50d0>, signals=[15]) 2024-03-25T18:22:32.070 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:22:32.076 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:22:32.076 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T18:22:32.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:32.183 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T18:22:32.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:32.271 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:22:32.278 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-03-25T18:22:32.295 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-03-25T18:22:32.295 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:32.352 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:32.353 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:32.353 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:32.353 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:32.353 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-03-25T18:22:32.373 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2024-03-25T18:22:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2024-03-25T18:22:32.390 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2024-03-25T18:22:32.445 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2024-03-25T18:22:32.445 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="9" 2024-03-25T18:22:32.445 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2024-03-25T18:22:32.445 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:22:32.445 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="9" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:22:32.446 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:22:32.446 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2024-03-25T18:22:32.464 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:22:32.473 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:22:32.477 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:22:32.478 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:32.480 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:22:32.500 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:22:32.506 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:22:32.506 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:32.536 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:22:32.626 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:22:32.680 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:22:32.694 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:22:32.694 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:22:32.694 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:32.694 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:22:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:32.710 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-03-25T18:22:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:22:32.711 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-03-25T18:22:32.766 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:32.766 INFO:teuthology.task.kernel:Running kernel on smithi076: 5.14.0-361.el9.x86_64 2024-03-25T18:22:32.766 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2024-03-25T18:22:32.769 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:22:32.769 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-361.el9.x86_64 2024-03-25T18:22:32.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-03-25T18:22:35.323 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - AppStream 7.6 MB/s | 6.7 MB 00:00 2024-03-25T18:22:35.342 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - AppStream 7.2 MB/s | 6.7 MB 00:00 2024-03-25T18:22:36.823 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-25T18:22:36.900 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-25T18:22:40.731 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-25T18:22:41.053 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:22:45.768 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:03 2024-03-25T18:22:46.514 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-25T18:22:53.437 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-25T18:22:54.301 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-25T18:22:54.875 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-25T18:22:56.181 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-03-25T18:22:56.738 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T18:22:58.373 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T18:23:01.602 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T18:23:03.590 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T18:23:03.793 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:23:03.793 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:23:03.910 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:23:03.915 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:23:03.915 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-03-25T18:23:03.915 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:23:03.915 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout:Install 4 Packages 2024-03-25T18:23:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:23:03.917 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 76 M 2024-03-25T18:23:03.917 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 123 M 2024-03-25T18:23:03.917 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:23:03.969 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:03.969 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:03.969 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:04.063 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:04.065 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:04.072 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:04.130 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:04.130 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:04.139 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:04.274 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:04.284 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:04.308 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:04.309 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:04.318 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:04.342 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:04.392 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:04.402 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:04.437 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:04.563 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:04.584 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:04.609 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:04.727 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:04.743 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:04.768 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:05.077 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:05.100 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:05.166 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:05.230 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:05.243 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:05.306 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:05.307 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:05.341 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:05.384 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.384 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:05.392 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.425 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.435 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.435 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.452 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.628 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 40 kB/s | 66 kB 00:01 2024-03-25T18:23:05.686 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.705 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:05.830 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:23:05.830 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:23:05.903 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:05.937 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:23:05.943 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:23:05.944 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-03-25T18:23:05.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:23:05.945 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 76 M 2024-03-25T18:23:05.945 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 123 M 2024-03-25T18:23:05.945 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:23:05.988 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:06.054 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.088 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:06.184 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:06.280 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.308 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.309 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.309 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.421 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.430 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.449 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:06.454 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:06.491 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:06.502 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:06.537 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 5.8 MB/s | 15 MB 00:02 2024-03-25T18:23:06.551 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:06.585 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:06.662 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:06.667 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:06.711 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:06.724 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:06.803 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:07.039 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:07.104 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:07.182 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:07.273 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:07.308 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:07.311 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:07.328 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:07.379 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 9.8 MB/s | 33 MB 00:03 2024-03-25T18:23:07.514 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:07.537 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:07.659 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:07.696 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:07.743 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:07.762 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:07.762 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:07.782 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:08.104 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 11 MB/s | 27 MB 00:02 2024-03-25T18:23:08.109 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:08.109 INFO:teuthology.orchestra.run.smithi076.stdout:Total 18 MB/s | 76 MB 00:04 2024-03-25T18:23:08.550 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): kernel-core-5.14.0-432.el9.x86_64.rpm 6.4 MB/s | 15 MB 00:02 2024-03-25T18:23:08.678 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:08.750 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:23:08.767 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:08.770 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:23:08.821 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:09.125 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 11 MB/s | 33 MB 00:02 2024-03-25T18:23:09.207 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:09.219 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:23:09.220 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:23:09.561 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:09.596 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:09.630 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:09.665 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:09.810 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:09.898 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:09.959 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:09.979 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:10.253 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:23:10.253 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:23:10.708 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-25T18:23:12.484 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:12.565 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:12.622 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:12.658 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:12.724 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:23:13.009 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:23:13.028 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:23:13.165 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:23:13.254 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:13.344 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:13.400 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:23:13.417 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:23:13.435 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:13.550 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): kernel-5.14.0-432.el9.x86_64.rpm 9.0 kB/s | 66 kB 00:07 2024-03-25T18:23:13.552 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:13.553 INFO:teuthology.orchestra.run.smithi050.stdout:Total 10 MB/s | 76 MB 00:07 2024-03-25T18:23:14.243 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:23:14.668 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:23:14.669 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:23:15.701 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:23:15.701 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:23:16.185 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:16.317 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:18.311 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:23:18.954 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:19.044 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:21.035 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:21.151 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:21.943 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:22.080 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:25.922 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:27.180 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:27.222 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:32.356 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:55.647 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:55.648 INFO:teuthology.orchestra.run.smithi076.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:23:55.648 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:23:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:56.129 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:56.130 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:23:56.130 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:23:56.130 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:23:56.542 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:23:56.542 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:23:56.542 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:23:56.543 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:56.543 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:56.543 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:23:56.543 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:23:56.543 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:23:56.543 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:23:56.884 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:57.002 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:57.002 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:57.002 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:23:57.002 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:23:57.002 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:23:57.002 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:23:57.003 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:23:57.003 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:23:57.083 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:23:57.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:23:57.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2024-03-25T18:23:57.117 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.06 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 68.el9 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Group : Unspecified 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 8264254 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:23:57.118 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : builder@centos.org 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:23:57.119 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:23:57.120 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:23:57.120 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:23:57.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:23:57.743 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2024-03-25T18:23:59.122 INFO:teuthology.orchestra.run.smithi076.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:23:59.152 INFO:teuthology.orchestra.run.smithi076.stderr:done 2024-03-25T18:23:59.155 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-03-25T18:23:59.170 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.92P7f04xEQ 2024-03-25T18:23:59.171 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.92P7f04xEQ 2024-03-25T18:23:59.238 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.92P7f04xEQ 2024-03-25T18:23:59.376 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.92P7f04xEQ is 6KB 2024-03-25T18:23:59.424 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.92P7f04xEQ 2024-03-25T18:23:59.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:23:59.502 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:23:59.503 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:23:59.503 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:23:59.503 INFO:teuthology.orchestra.run.smithi076.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:23:59.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:23:59.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2024-03-25T18:24:03.234 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:24:03.235 INFO:teuthology.orchestra.run.smithi050.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:24:03.235 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:24:03.282 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:24:03.691 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:24:03.691 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:24:03.691 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:24:03.691 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:24:04.153 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:24:04.478 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:24:04.631 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:24:04.632 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:24:04.632 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:24:04.632 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:24:04.632 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:24:04.632 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:24:04.632 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:24:04.633 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:24:04.715 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:24:04.716 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:24:04.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2024-03-25T18:24:04.753 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.06 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 68.el9 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Group : Unspecified 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 8264254 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:24:04.754 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : builder@centos.org 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:24:04.755 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:24:04.756 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:24:04.756 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:24:04.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:24:05.320 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-03-25T18:24:06.767 INFO:teuthology.orchestra.run.smithi050.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:24:06.810 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-03-25T18:24:06.812 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-03-25T18:24:06.827 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.XmaD10Rpkg 2024-03-25T18:24:06.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XmaD10Rpkg 2024-03-25T18:24:06.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.XmaD10Rpkg 2024-03-25T18:24:07.039 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.XmaD10Rpkg is 6KB 2024-03-25T18:24:07.088 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.XmaD10Rpkg 2024-03-25T18:24:07.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:24:07.170 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:24:07.171 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:24:07.171 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:24:07.171 INFO:teuthology.orchestra.run.smithi050.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:24:07.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:24:07.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-03-25T18:24:29.678 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:24:29.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:24:29.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:37.338 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:24:37.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-03-25T18:24:37.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:48.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-25T18:24:55.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-03-25T18:24:57.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:24:57.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:00.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-25T18:25:04.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-03-25T18:25:04.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:12.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:25:12.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:15.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-25T18:25:30.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:25:30.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:25:48.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2024-03-25T18:26:04.862 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:26:06.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:26:06.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:26:07.120 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-03-25T18:26:07.491 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:26:07.492 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:26:07.492 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2024-03-25T18:26:07.507 INFO:teuthology.orchestra.run.smithi076.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:26:07.508 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:26:07.508 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:26:07.508 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:26:08.509 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:26:08.510 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:26:08.876 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2024-03-25T18:26:08.912 DEBUG:teuthology.parallel:result is None 2024-03-25T18:26:16.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-03-25T18:26:16.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:26:17.206 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-03-25T18:26:17.596 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-03-25T18:26:17.596 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:26:17.596 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-03-25T18:26:17.612 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:26:17.612 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:26:17.612 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:26:17.612 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:26:18.613 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:26:18.613 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:26:18.689 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-03-25T18:26:18.725 DEBUG:teuthology.parallel:result is None 2024-03-25T18:26:18.726 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:26:18.733 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:26:18.733 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:26:18.735 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:26:18.753 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:26:18.759 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:26:18.767 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:26:18.767 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:26:18.795 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:26:18.828 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:26:18.835 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:26:18.835 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:26:18.861 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:26:18.891 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:26:18.903 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:26:18.904 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:26:18.918 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:26:18.920 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:26:18.927 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:26:18.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:26:18.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:26:18.991 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:26:19.003 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:26:19.004 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:26:19.018 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:26:19.049 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:26:19.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:26:19.167 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:26:19.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:26:19.291 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T18:26:19.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:26:19.358 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:26:19.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:26:19.457 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:26:19.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:26:19.588 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T18:26:19.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:26:19.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-03-25T18:26:19.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2024-03-25T18:26:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:26:19.744 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:26:20.157 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:26:20.165 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:26:20.165 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:26:20.176 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:26:20.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-03-25T18:26:20.244 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-03-25T18:26:20.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2024-03-25T18:26:20.307 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2024-03-25T18:26:20.309 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:26:20.309 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-03-25T18:26:20.341 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-03-25T18:26:20.341 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2024-03-25T18:26:20.361 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2024-03-25T18:26:20.361 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi076.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:26:20.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:26:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:20.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:26:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:20.437 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:26:20.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:26:20.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:26:20.529 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:26:20.539 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:26:20.555 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:26:20.556 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi076.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:30:38.756 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi076.front.sepia.ceph.com')] 2024-03-25T18:30:38.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-03-25T18:30:38.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:30:38.849 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-03-25T18:30:38.925 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-03-25T18:30:38.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:30:38.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:30:39.020 DEBUG:teuthology.orchestra.run.smithi076:> true 2024-03-25T18:30:39.105 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2024-03-25T18:30:39.106 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:30:39.116 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:30:39.116 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:30:39.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:30:39.119 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:30:39.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:30:39.149 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:30:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:30:39.177 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:30:39.194 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:30:39.197 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-03-25T18:30:39.211 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-03-25T18:30:39.224 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2024-03-25T18:30:39.229 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:30:39.237 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2024-03-25T18:30:39.244 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:30:39.254 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:30:39.270 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:30:39.299 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:30:39.302 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:30:39.302 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-03-25T18:30:39.302 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.302 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.302 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.302 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.324 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:30:39.328 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:30:39.329 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-03-25T18:30:39.329 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.329 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.329 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.329 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:30:39.329 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:30:39.337 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:30:39.337 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:30:39.337 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:30:39.337 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:30:39.343 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:30:39.343 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:30:39.343 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:30:39.344 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:30:39.346 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:30:39.346 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:30:39.490 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:30:39.490 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:30:39.493 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:30:39.493 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:30:39.835 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:30:39.835 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T18:30:39.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:30:39.865 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:30:39.865 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:30:39.890 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:30:39.890 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T18:30:39.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:30:39.921 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:30:39.921 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:30:39.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:30:39.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:30:40.015 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:30:40.073 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:30:40.082 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2024-03-25T18:30:40.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2024-03-25T18:30:40.140 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-03-25T18:30:40.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-03-25T18:30:40.330 INFO:teuthology.orchestra.run.smithi076.stdout:74 files removed 2024-03-25T18:30:40.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw 2024-03-25T18:30:40.388 INFO:teuthology.orchestra.run.smithi050.stdout:74 files removed 2024-03-25T18:30:40.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2024-03-25T18:30:41.599 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 87 kB/s | 82 kB 00:00 2024-03-25T18:30:41.897 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 70 kB/s | 82 kB 00:01 2024-03-25T18:30:42.416 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T18:30:42.712 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T18:30:43.283 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T18:30:43.604 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T18:30:43.785 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:30:44.073 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T18:30:45.253 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-25T18:30:45.581 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-25T18:30:49.132 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-25T18:30:49.745 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T18:30:51.240 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-25T18:30:54.849 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-25T18:30:58.919 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-25T18:31:00.356 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-03-25T18:31:01.600 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-25T18:31:02.400 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-25T18:31:04.031 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-25T18:31:05.485 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T18:31:06.500 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T18:31:08.953 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:31:08.960 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:31:08.961 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:31:08.962 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:31:08.963 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout:Install 30 Packages 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 2 Packages 2024-03-25T18:31:08.964 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:31:08.965 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 58 M 2024-03-25T18:31:08.965 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:31:10.575 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-25T18:31:13.133 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:31:13.146 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:31:13.146 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:31:13.146 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:31:13.146 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:31:13.147 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:31:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout:Install 30 Packages 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2024-03-25T18:31:13.149 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:31:13.150 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 58 M 2024-03-25T18:31:13.150 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:31:28.342 INFO:teuthology.orchestra.run.smithi076.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 519 kB/s | 9.7 MB 00:19 2024-03-25T18:31:28.525 INFO:teuthology.orchestra.run.smithi076.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 137 kB/s | 25 kB 00:00 2024-03-25T18:31:29.075 INFO:teuthology.orchestra.run.smithi076.stdout:(3/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 275 kB/s | 5.3 MB 00:19 2024-03-25T18:31:30.809 INFO:teuthology.orchestra.run.smithi076.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 288 kB/s | 498 kB 00:01 2024-03-25T18:31:31.159 INFO:teuthology.orchestra.run.smithi076.stdout:(5/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 282 kB/s | 742 kB 00:02 2024-03-25T18:31:31.326 INFO:teuthology.orchestra.run.smithi076.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 270 kB/s | 45 kB 00:00 2024-03-25T18:31:31.768 INFO:teuthology.orchestra.run.smithi076.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 297 kB/s | 131 kB 00:00 2024-03-25T18:31:31.916 INFO:teuthology.orchestra.run.smithi050.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 299 kB/s | 5.3 MB 00:18 2024-03-25T18:31:32.441 INFO:teuthology.orchestra.run.smithi050.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 48 kB/s | 25 kB 00:00 2024-03-25T18:31:32.443 INFO:teuthology.orchestra.run.smithi076.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 240 kB/s | 162 kB 00:00 2024-03-25T18:31:34.143 INFO:teuthology.orchestra.run.smithi076.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 188 kB/s | 320 kB 00:01 2024-03-25T18:31:34.549 INFO:teuthology.orchestra.run.smithi050.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 352 kB/s | 742 kB 00:02 2024-03-25T18:31:35.134 INFO:teuthology.orchestra.run.smithi076.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 300 kB/s | 298 kB 00:00 2024-03-25T18:31:35.593 INFO:teuthology.orchestra.run.smithi076.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 217 kB/s | 99 kB 00:00 2024-03-25T18:31:35.702 INFO:teuthology.orchestra.run.smithi076.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T18:31:35.744 INFO:teuthology.orchestra.run.smithi076.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T18:31:35.777 INFO:teuthology.orchestra.run.smithi076.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:31:35.836 INFO:teuthology.orchestra.run.smithi076.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:31:35.877 INFO:teuthology.orchestra.run.smithi076.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:31:35.911 INFO:teuthology.orchestra.run.smithi076.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T18:31:35.946 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:35.983 INFO:teuthology.orchestra.run.smithi050.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 348 kB/s | 498 kB 00:01 2024-03-25T18:31:36.110 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:36.216 INFO:teuthology.orchestra.run.smithi050.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 442 kB/s | 9.7 MB 00:22 2024-03-25T18:31:36.400 INFO:teuthology.orchestra.run.smithi050.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 246 kB/s | 45 kB 00:00 2024-03-25T18:31:36.475 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:36.542 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:36.683 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:36.700 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:36.734 INFO:teuthology.orchestra.run.smithi050.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 393 kB/s | 131 kB 00:00 2024-03-25T18:31:36.754 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:36.819 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:36.943 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:37.082 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:37.168 INFO:teuthology.orchestra.run.smithi050.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 374 kB/s | 162 kB 00:00 2024-03-25T18:31:37.171 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:37.224 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:37.246 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:37.378 INFO:teuthology.orchestra.run.smithi076.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 150 kB/s | 221 kB 00:01 2024-03-25T18:31:37.395 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:37.476 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:37.557 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:37.591 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:37.629 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:37.649 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:37.715 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:37.748 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:37.780 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:37.923 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:37.976 INFO:teuthology.orchestra.run.smithi050.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 396 kB/s | 320 kB 00:00 2024-03-25T18:31:37.999 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:38.054 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:38.071 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:38.112 INFO:teuthology.orchestra.run.smithi076.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 49 kB/s | 36 kB 00:00 2024-03-25T18:31:38.128 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:38.209 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:38.290 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:38.322 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:38.355 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:38.372 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:38.432 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:38.464 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:38.496 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:38.637 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:38.726 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:31:38.783 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:38.799 INFO:teuthology.orchestra.run.smithi076.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:38.826 INFO:teuthology.orchestra.run.smithi050.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 350 kB/s | 298 kB 00:00 2024-03-25T18:31:38.837 INFO:teuthology.orchestra.run.smithi076.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-03-25T18:31:39.104 INFO:teuthology.orchestra.run.smithi076.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T18:31:39.218 INFO:teuthology.orchestra.run.smithi050.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 253 kB/s | 99 kB 00:00 2024-03-25T18:31:39.327 INFO:teuthology.orchestra.run.smithi050.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-25T18:31:39.377 INFO:teuthology.orchestra.run.smithi050.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:31:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T18:31:39.469 INFO:teuthology.orchestra.run.smithi050.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:31:39.511 INFO:teuthology.orchestra.run.smithi050.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:31:39.537 INFO:teuthology.orchestra.run.smithi050.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T18:31:39.563 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:39.621 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:39.656 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:39.734 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:39.878 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:39.977 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:40.109 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:40.112 INFO:teuthology.orchestra.run.smithi076.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-03-25T18:31:40.269 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:40.271 INFO:teuthology.orchestra.run.smithi076.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-03-25T18:31:40.388 INFO:teuthology.orchestra.run.smithi076.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-25T18:31:40.447 INFO:teuthology.orchestra.run.smithi076.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T18:31:40.623 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:40.681 INFO:teuthology.orchestra.run.smithi076.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-03-25T18:31:40.779 INFO:teuthology.orchestra.run.smithi050.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 178 kB/s | 221 kB 00:01 2024-03-25T18:31:40.796 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:40.853 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:40.870 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:40.903 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:40.930 INFO:teuthology.orchestra.run.smithi076.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 767 kB/s | 192 kB 00:00 2024-03-25T18:31:40.936 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:40.969 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:41.006 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:41.086 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:41.221 INFO:teuthology.orchestra.run.smithi050.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 81 kB/s | 36 kB 00:00 2024-03-25T18:31:41.238 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:41.294 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:31:41.297 INFO:teuthology.orchestra.run.smithi076.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 4.3 MB/s | 1.6 MB 00:00 2024-03-25T18:31:41.311 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:31:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:41.382 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:31:41.417 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:41.456 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:31:41.550 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:41.658 INFO:teuthology.orchestra.run.smithi050.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:31:41.829 INFO:teuthology.orchestra.run.smithi050.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 78 kB/s | 47 kB 00:00 2024-03-25T18:31:42.238 INFO:teuthology.orchestra.run.smithi050.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 757 kB/s | 309 kB 00:00 2024-03-25T18:31:44.047 INFO:teuthology.orchestra.run.smithi050.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 4.4 MB 00:01 2024-03-25T18:31:44.238 INFO:teuthology.orchestra.run.smithi050.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 136 kB/s | 26 kB 00:00 2024-03-25T18:31:44.322 INFO:teuthology.orchestra.run.smithi050.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 584 kB/s | 49 kB 00:00 2024-03-25T18:31:44.381 INFO:teuthology.orchestra.run.smithi050.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T18:31:44.601 INFO:teuthology.orchestra.run.smithi050.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 838 kB 00:00 2024-03-25T18:31:44.714 INFO:teuthology.orchestra.run.smithi076.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 376 kB/s | 5.1 MB 00:13 2024-03-25T18:31:44.715 INFO:teuthology.orchestra.run.smithi050.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-25T18:31:45.057 INFO:teuthology.orchestra.run.smithi050.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-03-25T18:31:46.581 INFO:teuthology.orchestra.run.smithi076.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 654 kB/s | 3.4 MB 00:05 2024-03-25T18:31:49.739 INFO:teuthology.orchestra.run.smithi076.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 502 kB/s | 20 MB 00:40 2024-03-25T18:31:50.481 INFO:teuthology.orchestra.run.smithi050.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 360 kB/s | 5.1 MB 00:14 2024-03-25T18:31:55.305 INFO:teuthology.orchestra.run.smithi076.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 303 kB/s | 3.1 MB 00:10 2024-03-25T18:31:55.312 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:55.312 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.2 MB/s | 58 MB 00:46 2024-03-25T18:31:55.342 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:31:55.381 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:31:55.381 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:31:55.861 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:31:55.862 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:31:56.398 INFO:teuthology.orchestra.run.smithi050.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 542 kB/s | 3.1 MB 00:05 2024-03-25T18:31:56.757 INFO:teuthology.orchestra.run.smithi050.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 295 kB/s | 3.4 MB 00:11 2024-03-25T18:31:57.333 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:31:57.414 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:31:57.443 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:31:57.469 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:31:57.491 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:57.508 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:31:57.651 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:57.705 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:31:58.024 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:31:58.087 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:58.161 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:31:58.248 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:31:58.272 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:58.359 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:31:58.387 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:31:58.439 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:31:58.507 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:31:58.532 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:58.595 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:31:58.631 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:31:58.728 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:31:58.773 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:31:58.831 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:31:58.862 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:31:58.891 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:31:58.924 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:31:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:31:59.262 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:59.335 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:31:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:31:59.406 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:31:59.856 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:31:59.881 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:31:59.953 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:32:00.021 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:32:00.050 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:32:02.232 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:32:02.271 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:32:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:32:02.865 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:32:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:32:02.956 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:32:02.956 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:32:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:32:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:08.931 INFO:teuthology.orchestra.run.smithi050.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 368 kB/s | 20 MB 00:55 2024-03-25T18:32:08.937 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:08.937 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.0 MB/s | 58 MB 00:55 2024-03-25T18:32:08.967 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:32:09.009 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:32:09.009 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:32:09.512 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:32:09.513 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:32:11.187 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:32:11.271 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:32:11.302 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:32:11.327 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:32:11.350 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:32:11.367 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:32:11.542 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:32:11.598 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:32:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:32:12.013 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:32:12.089 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:32:12.165 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:12.165 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2024-03-25T18:32:12.165 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2024-03-25T18:32:12.166 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2024-03-25T18:32:12.166 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2024-03-25T18:32:12.166 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2024-03-25T18:32:12.166 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /root 2024-03-25T18:32:12.166 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2024-03-25T18:32:12.166 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:32:12.179 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:32:12.206 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:32:12.252 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:32:12.281 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:32:12.326 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:32:12.395 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:32:12.417 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:32:12.464 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:32:12.499 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:32:12.609 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:32:12.650 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:32:12.690 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:32:12.723 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:32:12.752 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:32:12.785 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:32:13.079 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:32:13.103 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:32:13.103 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:32:13.103 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:32:13.103 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:32:13.104 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:32:13.104 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:32:13.105 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:32:13.117 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:32:13.148 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:32:13.182 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:32:13.182 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:13.222 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:32:13.253 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:32:13.296 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:32:13.778 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:32:13.809 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:32:13.881 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:32:13.951 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:32:13.987 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:32:14.364 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:32:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:32:14.365 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:32:14.366 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:32:14.367 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:32:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:32:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:32:14.368 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:32:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:32:14.369 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:32:15.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.465 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:32:15.466 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:32:15.467 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:32:15.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-test 2024-03-25T18:32:16.232 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:32:16.267 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:32:16.379 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:10 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:32:16.539 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:32:16.539 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:32:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2024-03-25T18:32:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:32:16.541 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 48 M 2024-03-25T18:32:16.541 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 204 M 2024-03-25T18:32:16.541 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:32:16.887 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:32:16.928 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:32:17.061 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 893 kB/s | 217 kB 00:00 2024-03-25T18:32:17.103 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-25T18:32:17.128 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T18:32:17.154 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:32:17.154 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:32:17.154 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:32:17.154 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:32:17.222 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:17.328 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-03-25T18:32:17.528 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 230 kB/s | 163 kB 00:00 2024-03-25T18:32:27.021 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-03-25T18:32:27.022 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:32:28.063 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:32:28.088 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:32:28.089 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:32:28.089 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:32:28.089 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:32:28.089 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:32:28.089 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:32:28.090 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:32:28.182 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:32:28.182 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:32:29.324 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:32:29.325 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:32:29.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:32:29.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:32:29.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:32:29.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:32:29.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:32:29.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:32:29.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:32:29.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:32:29.327 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:32:30.326 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:32:30.326 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:32:30.326 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-03-25T18:32:30.326 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:32:30.327 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:32:30.328 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:32:30.329 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:32:30.329 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:32:30.329 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:32:30.329 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:32:30.329 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:32:30.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2024-03-25T18:32:31.186 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:21 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:32:31.351 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:32:31.351 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:32:31.351 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:32:31.352 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-03-25T18:32:31.353 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:32:31.353 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 48 M 2024-03-25T18:32:31.353 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 204 M 2024-03-25T18:32:31.353 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:32:31.663 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 795 kB/s | 217 kB 00:00 2024-03-25T18:32:31.713 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.1 MB/s | 300 kB 00:00 2024-03-25T18:32:31.747 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T18:32:31.930 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T18:32:32.914 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 107 kB/s | 163 kB 00:01 2024-03-25T18:33:54.910 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 493 kB/s | 47 MB 01:38 2024-03-25T18:33:54.912 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:54.912 INFO:teuthology.orchestra.run.smithi076.stdout:Total 502 kB/s | 48 MB 01:38 2024-03-25T18:33:54.923 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:33:54.943 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:33:54.943 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:33:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:33:55.324 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:33:55.917 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:33:55.985 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:33:56.025 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:33:56.066 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:33:56.117 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:33:56.146 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:34:00.618 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:34:00.696 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:34:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:34:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:34:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:34:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:34:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:34:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:34:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:03.473 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:34:03.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph 2024-03-25T18:34:04.375 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:58 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:34:04.527 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:34:04.529 INFO:teuthology.orchestra.run.smithi076.stdout:========================================================================================= 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout:========================================================================================= 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:34:04.530 INFO:teuthology.orchestra.run.smithi076.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:34:04.531 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:34:04.532 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout:========================================================================================= 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout:Install 36 Packages 2024-03-25T18:34:04.533 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:04.534 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 30 M 2024-03-25T18:34:04.534 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 107 M 2024-03-25T18:34:04.534 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:34:05.196 INFO:teuthology.orchestra.run.smithi076.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T18:34:08.513 INFO:teuthology.orchestra.run.smithi076.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 414 kB/s | 1.5 MB 00:03 2024-03-25T18:34:11.705 INFO:teuthology.orchestra.run.smithi076.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 316 kB/s | 2.1 MB 00:06 2024-03-25T18:34:12.288 INFO:teuthology.orchestra.run.smithi076.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 424 kB/s | 247 kB 00:00 2024-03-25T18:34:12.413 INFO:teuthology.orchestra.run.smithi076.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:34:12.455 INFO:teuthology.orchestra.run.smithi076.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:34:12.481 INFO:teuthology.orchestra.run.smithi076.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:34:12.506 INFO:teuthology.orchestra.run.smithi076.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T18:34:12.531 INFO:teuthology.orchestra.run.smithi076.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:34:12.573 INFO:teuthology.orchestra.run.smithi076.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:34:12.648 INFO:teuthology.orchestra.run.smithi076.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:34:12.682 INFO:teuthology.orchestra.run.smithi076.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:34:12.724 INFO:teuthology.orchestra.run.smithi076.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:34:12.766 INFO:teuthology.orchestra.run.smithi076.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:34:13.183 INFO:teuthology.orchestra.run.smithi076.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 300 kB/s | 125 kB 00:00 2024-03-25T18:34:13.316 INFO:teuthology.orchestra.run.smithi076.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-25T18:34:13.650 INFO:teuthology.orchestra.run.smithi076.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 453 kB/s | 151 kB 00:00 2024-03-25T18:34:13.742 INFO:teuthology.orchestra.run.smithi076.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 475 kB/s | 43 kB 00:00 2024-03-25T18:34:13.842 INFO:teuthology.orchestra.run.smithi076.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T18:34:13.968 INFO:teuthology.orchestra.run.smithi076.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-25T18:34:14.034 INFO:teuthology.orchestra.run.smithi076.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-25T18:34:14.101 INFO:teuthology.orchestra.run.smithi076.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-25T18:34:14.168 INFO:teuthology.orchestra.run.smithi076.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T18:34:14.236 INFO:teuthology.orchestra.run.smithi076.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 290 kB/s | 19 kB 00:00 2024-03-25T18:34:14.303 INFO:teuthology.orchestra.run.smithi076.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T18:34:14.386 INFO:teuthology.orchestra.run.smithi076.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 556 kB/s | 46 kB 00:00 2024-03-25T18:34:14.453 INFO:teuthology.orchestra.run.smithi076.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-25T18:34:14.545 INFO:teuthology.orchestra.run.smithi076.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-25T18:34:14.604 INFO:teuthology.orchestra.run.smithi076.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-25T18:34:14.688 INFO:teuthology.orchestra.run.smithi076.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-25T18:34:14.763 INFO:teuthology.orchestra.run.smithi076.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 478 kB/s | 36 kB 00:00 2024-03-25T18:34:14.838 INFO:teuthology.orchestra.run.smithi076.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T18:34:14.930 INFO:teuthology.orchestra.run.smithi076.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.6 MB/s | 427 kB 00:00 2024-03-25T18:34:14.989 INFO:teuthology.orchestra.run.smithi076.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-25T18:34:16.680 INFO:teuthology.orchestra.run.smithi076.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 418 kB/s | 4.7 MB 00:11 2024-03-25T18:34:37.925 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 383 kB/s | 47 MB 02:06 2024-03-25T18:34:37.928 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:37.928 INFO:teuthology.orchestra.run.smithi050.stdout:Total 390 kB/s | 48 MB 02:06 2024-03-25T18:34:37.939 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:34:37.960 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:34:37.960 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:34:38.357 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:34:38.358 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:34:38.968 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:34:39.034 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:34:39.077 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:34:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:34:39.159 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:34:39.187 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:34:44.015 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:34:44.089 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:34:44.811 INFO:teuthology.orchestra.run.smithi076.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 473 kB/s | 17 MB 00:36 2024-03-25T18:34:44.815 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:44.816 INFO:teuthology.orchestra.run.smithi076.stdout:Total 754 kB/s | 30 MB 00:40 2024-03-25T18:34:44.838 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:34:44.879 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:34:44.879 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:34:45.305 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:34:45.305 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:34:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:34:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:34:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:34:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:34:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:34:46.271 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:34:46.626 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:34:46.626 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:34:46.627 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:34:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:34:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:34:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:34:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:34:46.827 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:34:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:34:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:34:46.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2024-03-25T18:34:46.988 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:34:47.066 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:34:47.108 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:34:47.179 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:34:47.234 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:34:47.306 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:34:47.354 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:34:47.409 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:34:47.478 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:34:47.602 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:34:47.644 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:34:47.647 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2024-03-25T18:34:47.647 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:34:47.647 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2024-03-25T18:34:47.647 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:34:47.647 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:34:47.648 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:34:47.649 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout:========================================================================================= 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout:Install 36 Packages 2024-03-25T18:34:47.650 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:34:47.651 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 30 M 2024-03-25T18:34:47.651 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 107 M 2024-03-25T18:34:47.651 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:34:47.651 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:34:47.690 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:34:47.724 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:34:47.755 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:34:47.830 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:34:47.865 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:47.898 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:34:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:34:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:34:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:34:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:34:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:34:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:34:48.339 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:34:48.464 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:48.623 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:34:48.652 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:48.662 INFO:teuthology.orchestra.run.smithi050.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:34:48.678 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:34:48.678 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:48.678 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:34:48.678 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:48.678 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:34:48.678 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:50.179 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:50.207 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:34:50.207 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:50.207 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:34:50.207 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:50.207 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:34:50.207 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:50.663 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:50.687 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:34:50.687 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:50.688 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:34:50.688 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:50.688 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:34:50.688 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:50.905 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:34:50.929 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:50.929 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:34:50.929 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:50.929 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:34:50.929 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:50.994 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:52.329 INFO:teuthology.orchestra.run.smithi050.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 370 kB/s | 1.5 MB 00:04 2024-03-25T18:34:52.753 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:34:52.753 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:34:52.754 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:34:52.755 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:34:52.756 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:34:52.756 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:34:52.757 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:34:53.254 INFO:teuthology.orchestra.run.smithi050.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 431 kB/s | 2.1 MB 00:05 2024-03-25T18:34:53.615 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:34:53.615 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:53.615 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:34:53.615 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:34:53.616 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:34:53.617 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:53.618 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:34:53.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-base 2024-03-25T18:34:54.095 INFO:teuthology.orchestra.run.smithi050.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 294 kB/s | 247 kB 00:00 2024-03-25T18:34:54.212 INFO:teuthology.orchestra.run.smithi050.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T18:34:54.254 INFO:teuthology.orchestra.run.smithi050.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:34:54.279 INFO:teuthology.orchestra.run.smithi050.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:34:54.305 INFO:teuthology.orchestra.run.smithi050.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T18:34:54.330 INFO:teuthology.orchestra.run.smithi050.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:34:54.372 INFO:teuthology.orchestra.run.smithi050.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:34:54.439 INFO:teuthology.orchestra.run.smithi050.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:34:54.472 INFO:teuthology.orchestra.run.smithi050.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:34:54.506 INFO:teuthology.orchestra.run.smithi050.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T18:34:54.539 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:03:48 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:34:54.540 INFO:teuthology.orchestra.run.smithi050.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T18:34:54.636 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:54.690 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:34:54.690 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:34:54.691 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:34:54.781 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install cephadm 2024-03-25T18:34:54.965 INFO:teuthology.orchestra.run.smithi050.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 294 kB/s | 125 kB 00:00 2024-03-25T18:34:55.007 INFO:teuthology.orchestra.run.smithi050.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 536 kB/s | 22 kB 00:00 2024-03-25T18:34:55.183 INFO:teuthology.orchestra.run.smithi050.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 863 kB/s | 151 kB 00:00 2024-03-25T18:34:55.216 INFO:teuthology.orchestra.run.smithi050.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-03-25T18:34:55.283 INFO:teuthology.orchestra.run.smithi050.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-25T18:34:55.342 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:03:49 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:34:55.350 INFO:teuthology.orchestra.run.smithi050.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.3 MB/s | 358 kB 00:00 2024-03-25T18:34:55.383 INFO:teuthology.orchestra.run.smithi050.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 322 kB/s | 11 kB 00:00 2024-03-25T18:34:55.417 INFO:teuthology.orchestra.run.smithi050.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 531 kB/s | 18 kB 00:00 2024-03-25T18:34:55.451 INFO:teuthology.orchestra.run.smithi050.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 693 kB/s | 23 kB 00:00 2024-03-25T18:34:55.484 INFO:teuthology.orchestra.run.smithi050.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-03-25T18:34:55.495 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:34:55.495 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 747 k 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 754 k 2024-03-25T18:34:55.496 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:34:55.526 INFO:teuthology.orchestra.run.smithi050.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 472 kB/s | 20 kB 00:00 2024-03-25T18:34:55.568 INFO:teuthology.orchestra.run.smithi050.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-25T18:34:55.610 INFO:teuthology.orchestra.run.smithi050.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.9 MB/s | 79 kB 00:00 2024-03-25T18:34:55.660 INFO:teuthology.orchestra.run.smithi050.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-25T18:34:55.702 INFO:teuthology.orchestra.run.smithi050.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-03-25T18:34:55.744 INFO:teuthology.orchestra.run.smithi050.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-25T18:34:55.786 INFO:teuthology.orchestra.run.smithi050.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 858 kB/s | 36 kB 00:00 2024-03-25T18:34:55.837 INFO:teuthology.orchestra.run.smithi050.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-25T18:34:55.887 INFO:teuthology.orchestra.run.smithi050.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.4 MB/s | 427 kB 00:00 2024-03-25T18:34:55.929 INFO:teuthology.orchestra.run.smithi050.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 482 kB/s | 20 kB 00:00 2024-03-25T18:34:57.818 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 322 kB/s | 747 kB 00:02 2024-03-25T18:34:57.818 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:57.819 INFO:teuthology.orchestra.run.smithi076.stdout:Total 322 kB/s | 747 kB 00:02 2024-03-25T18:34:57.819 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:34:57.822 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:34:57.822 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:34:57.828 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:34:57.828 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:34:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:34:58.386 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:58.486 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:58.848 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:34:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:59.132 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:34:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:59.132 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:34:59.132 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:34:59.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:34:59.944 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:03:53 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:00.098 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:00.099 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:00.099 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:35:00.099 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:00.099 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:00.099 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:35:00.099 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:00.099 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:00.100 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:00.100 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:35:00.100 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:00.100 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 145 k 2024-03-25T18:35:00.100 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 436 k 2024-03-25T18:35:00.100 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:01.063 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 151 kB/s | 145 kB 00:00 2024-03-25T18:35:01.064 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:01.064 INFO:teuthology.orchestra.run.smithi076.stdout:Total 151 kB/s | 145 kB 00:00 2024-03-25T18:35:01.065 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:01.073 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:01.073 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:01.129 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:01.129 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:01.270 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:01.347 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:01.372 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:01.372 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:01.372 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:35:01.373 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:02.079 INFO:teuthology.orchestra.run.smithi050.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 358 kB/s | 4.7 MB 00:13 2024-03-25T18:35:02.374 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:02.374 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:02.374 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:02.374 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:02.374 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:02.374 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:02.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr 2024-03-25T18:35:03.203 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:03:57 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:03.303 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:03.355 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:03.356 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:03.357 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:03.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:35:04.005 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:03:58 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:04.154 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:04.155 INFO:teuthology.orchestra.run.smithi076.stdout:====================================================================================== 2024-03-25T18:35:04.155 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:35:04.155 INFO:teuthology.orchestra.run.smithi076.stdout:====================================================================================== 2024-03-25T18:35:04.155 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:35:04.156 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout:====================================================================================== 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout:Install 12 Packages 2024-03-25T18:35:04.157 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:04.158 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 6.9 M 2024-03-25T18:35:04.158 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 26 M 2024-03-25T18:35:04.158 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:04.734 INFO:teuthology.orchestra.run.smithi076.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 68 kB/s | 31 kB 00:00 2024-03-25T18:35:04.817 INFO:teuthology.orchestra.run.smithi076.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 28 kB/s | 15 kB 00:00 2024-03-25T18:35:04.934 INFO:teuthology.orchestra.run.smithi076.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T18:35:04.960 INFO:teuthology.orchestra.run.smithi076.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T18:35:05.344 INFO:teuthology.orchestra.run.smithi076.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-25T18:35:05.485 INFO:teuthology.orchestra.run.smithi076.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-25T18:35:05.794 INFO:teuthology.orchestra.run.smithi076.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-03-25T18:35:05.844 INFO:teuthology.orchestra.run.smithi076.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.7 MB/s | 2.0 MB 00:00 2024-03-25T18:35:05.944 INFO:teuthology.orchestra.run.smithi076.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-03-25T18:35:06.045 INFO:teuthology.orchestra.run.smithi076.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T18:35:06.070 INFO:teuthology.orchestra.run.smithi076.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 524 kB/s | 144 kB 00:00 2024-03-25T18:35:06.820 INFO:teuthology.orchestra.run.smithi076.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 715 kB/s | 1.8 MB 00:02 2024-03-25T18:35:06.823 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:06.823 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.6 MB/s | 6.9 MB 00:02 2024-03-25T18:35:06.843 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:06.862 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:06.862 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:07.040 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:07.040 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:07.599 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:07.679 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:35:07.714 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:35:07.761 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:35:07.791 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:35:07.899 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:35:08.036 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:35:08.119 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:35:08.163 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:35:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:08.223 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:35:08.466 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:35:08.498 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:35:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:35:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:35:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:35:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:35:09.278 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:35:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:35:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:35:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:35:09.279 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:35:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:35:09.630 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:09.630 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:35:09.631 INFO:teuthology.orchestra.run.smithi076.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:35:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:35:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:35:09.632 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:09.632 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:09.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:35:10.438 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:10.587 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:35:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout:Install 12 Packages 2024-03-25T18:35:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:10.591 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 37 M 2024-03-25T18:35:10.591 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 209 M 2024-03-25T18:35:10.591 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:10.902 INFO:teuthology.orchestra.run.smithi076.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 32 kB 00:00 2024-03-25T18:35:10.952 INFO:teuthology.orchestra.run.smithi076.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 655 kB/s | 17 kB 00:00 2024-03-25T18:35:10.977 INFO:teuthology.orchestra.run.smithi076.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T18:35:11.136 INFO:teuthology.orchestra.run.smithi076.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:35:11.278 INFO:teuthology.orchestra.run.smithi076.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T18:35:11.328 INFO:teuthology.orchestra.run.smithi076.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-25T18:35:11.354 INFO:teuthology.orchestra.run.smithi076.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.8 MB/s | 217 kB 00:00 2024-03-25T18:35:11.529 INFO:teuthology.orchestra.run.smithi076.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T18:35:11.671 INFO:teuthology.orchestra.run.smithi076.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-25T18:35:11.696 INFO:teuthology.orchestra.run.smithi076.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-25T18:35:12.188 INFO:teuthology.orchestra.run.smithi076.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 475 kB/s | 245 kB 00:00 2024-03-25T18:35:20.711 INFO:teuthology.orchestra.run.smithi050.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 605 kB/s | 17 MB 00:28 2024-03-25T18:35:20.715 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:20.715 INFO:teuthology.orchestra.run.smithi050.stdout:Total 919 kB/s | 30 MB 00:33 2024-03-25T18:35:20.740 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:35:20.786 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:35:20.786 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:35:21.217 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:35:21.217 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:35:21.287 INFO:teuthology.orchestra.run.smithi076.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 719 kB/s | 7.4 MB 00:10 2024-03-25T18:35:21.289 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:21.290 INFO:teuthology.orchestra.run.smithi076.stdout:Total 3.5 MB/s | 37 MB 00:10 2024-03-25T18:35:21.332 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:21.357 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:21.357 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:21.709 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:21.710 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:22.429 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:22.493 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:35:22.524 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:35:22.627 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:22.661 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:35:22.671 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:35:22.730 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:35:22.773 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:35:22.825 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:35:22.839 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:22.877 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:22.907 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:35:22.948 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:35:22.960 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:35:22.981 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:35:23.077 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:35:23.279 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:35:23.326 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:35:23.400 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:35:23.452 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:35:23.490 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:35:23.548 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:35:23.610 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:35:23.611 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:35:23.723 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:35:23.807 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:35:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:35:23.899 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:35:23.933 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:35:23.964 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:35:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:35:24.073 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:35:24.107 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:35:24.147 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:35:24.182 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:35:24.216 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:35:24.310 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:35:24.348 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:35:24.423 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:35:24.565 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:35:24.707 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:35:24.880 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:35:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:35:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:35:24.936 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:24.936 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:35:24.936 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:35:24.936 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:35:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:24.950 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:35:25.974 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:35:26.002 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:26.471 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:35:26.499 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:35:26.499 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:26.499 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:35:26.500 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:35:26.500 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:35:26.500 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:26.964 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:35:26.987 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:35:26.988 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:26.988 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:35:26.988 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:35:26.988 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:35:26.988 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:27.206 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:35:27.229 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:35:27.229 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:27.229 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:35:27.229 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:35:27.229 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:35:27.229 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:27.297 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:35:28.526 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:35:28.527 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:35:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:35:28.925 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:28.925 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:28.926 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:35:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:35:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:35:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:35:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:35:28.955 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:35:28.956 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:35:28.957 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:35:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:35:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:35:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:35:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:35:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:35:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:35:28.958 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:35:29.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:35:29.902 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:30.050 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:35:30.052 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout:Install 13 Packages 2024-03-25T18:35:30.053 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:35:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:30.054 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:35:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:35:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:35:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 2.0 M 2024-03-25T18:35:30.057 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 26 M 2024-03-25T18:35:30.058 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:30.253 INFO:teuthology.orchestra.run.smithi076.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2024-03-25T18:35:30.320 INFO:teuthology.orchestra.run.smithi076.stdout:(2/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T18:35:30.345 INFO:teuthology.orchestra.run.smithi076.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 82 kB/s | 18 kB 00:00 2024-03-25T18:35:30.370 INFO:teuthology.orchestra.run.smithi076.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-25T18:35:30.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2024-03-25T18:35:30.395 INFO:teuthology.orchestra.run.smithi076.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:35:30.421 INFO:teuthology.orchestra.run.smithi076.stdout:(6/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 176 kB/s | 52 kB 00:00 2024-03-25T18:35:30.446 INFO:teuthology.orchestra.run.smithi076.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 570 kB/s | 43 kB 00:00 2024-03-25T18:35:30.613 INFO:teuthology.orchestra.run.smithi076.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-03-25T18:35:30.638 INFO:teuthology.orchestra.run.smithi076.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-25T18:35:30.772 INFO:teuthology.orchestra.run.smithi076.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 623 kB/s | 203 kB 00:00 2024-03-25T18:35:30.805 INFO:teuthology.orchestra.run.smithi076.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.3 MB/s | 1.0 MB 00:00 2024-03-25T18:35:30.830 INFO:teuthology.orchestra.run.smithi076.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-25T18:35:30.855 INFO:teuthology.orchestra.run.smithi076.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-25T18:35:30.860 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:30.860 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-25T18:35:30.860 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:30.873 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:30.873 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:30.955 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:35:30.963 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:30.963 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:31.056 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:31.110 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:35:31.112 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:35:31.112 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:35:31.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2024-03-25T18:35:31.487 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:31.656 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:35:31.718 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:35:31.745 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:35:31.781 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:35:31.818 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:35:31.860 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:35:31.900 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 747 k 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 754 k 2024-03-25T18:35:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:35:31.954 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:35:32.055 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:35:32.102 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:35:32.832 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:35:32.892 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:35:32.926 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:35:32.974 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:35:32.996 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:33.155 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 596 kB/s | 747 kB 00:01 2024-03-25T18:35:33.156 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:33.157 INFO:teuthology.orchestra.run.smithi050.stdout:Total 595 kB/s | 747 kB 00:01 2024-03-25T18:35:33.157 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:35:33.164 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:35:33.164 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:35:33.170 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:35:33.170 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:35:33.249 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:35:33.668 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:33.763 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:35:33.791 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:35:33.792 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:35:33.792 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:35:34.141 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:34.153 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:35:34.153 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:34.153 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:34.153 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:34.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:35:34.153 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:35:34.154 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:34.155 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:34.420 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:35:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:34.421 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:35:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:34.421 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:34.421 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:35:34.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:35:34.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:35:35.018 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:29 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:35.169 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:35.170 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:35:35.170 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:35:35.170 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:35:35.170 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2024-03-25T18:35:35.171 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:35.172 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 6.8 M 2024-03-25T18:35:35.172 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 33 M 2024-03-25T18:35:35.172 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:35.188 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:25 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:35:35.343 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:35:35.343 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:35:35.344 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:35.345 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 145 k 2024-03-25T18:35:35.345 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 436 k 2024-03-25T18:35:35.345 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:35:35.619 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-25T18:35:35.802 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.9 MB/s | 548 kB 00:00 2024-03-25T18:35:35.844 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-03-25T18:35:35.886 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-25T18:35:35.953 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 419 kB/s | 248 kB 00:00 2024-03-25T18:35:36.128 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 186 kB/s | 145 kB 00:00 2024-03-25T18:35:36.129 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:36.129 INFO:teuthology.orchestra.run.smithi050.stdout:Total 185 kB/s | 145 kB 00:00 2024-03-25T18:35:36.129 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:35:36.139 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:35:36.139 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:35:36.186 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 167 kB/s | 138 kB 00:00 2024-03-25T18:35:36.188 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:36.188 INFO:teuthology.orchestra.run.smithi076.stdout:Total 6.7 MB/s | 6.8 MB 00:01 2024-03-25T18:35:36.195 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:35:36.195 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:35:36.201 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:36.211 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:36.211 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:36.300 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:36.300 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:36.342 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:35:36.412 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:36.439 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:36.439 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:36.439 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:35:36.439 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:36.598 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:36.778 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:35:36.821 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:35:37.359 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:35:37.442 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:35:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:35:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:37.454 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:35:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:37.454 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:37.454 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:35:37.503 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:35:37.526 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:37.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2024-03-25T18:35:38.286 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:35:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:35:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:35:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:35:38.335 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:35:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:35:38.336 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:35:38.387 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:38.442 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:35:38.443 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:35:38.443 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:35:38.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:35:38.625 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:35:38.625 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:38.625 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:38.625 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:38.625 INFO:teuthology.orchestra.run.smithi076.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:35:38.625 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:35:38.626 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:35:38.626 INFO:teuthology.orchestra.run.smithi076.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:35:38.626 INFO:teuthology.orchestra.run.smithi076.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:35:38.626 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:38.626 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:38.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-fuse 2024-03-25T18:35:39.080 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:29 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:35:39.236 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:35:39.237 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-03-25T18:35:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:35:39.237 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-03-25T18:35:39.237 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:35:39.237 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:35:39.238 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:35:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:39.239 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:35:39.239 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-03-25T18:35:39.239 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-03-25T18:35:39.239 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:39.239 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 6.9 M 2024-03-25T18:35:39.240 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-03-25T18:35:39.240 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:35:39.459 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:33 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:39.609 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:39.611 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:39.612 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:39.612 INFO:teuthology.orchestra.run.smithi076.stdout:Install 2 Packages 2024-03-25T18:35:39.612 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:39.612 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 964 k 2024-03-25T18:35:39.612 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 2.8 M 2024-03-25T18:35:39.612 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:39.747 INFO:teuthology.orchestra.run.smithi076.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 582 kB/s | 79 kB 00:00 2024-03-25T18:35:39.949 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 28 kB/s | 15 kB 00:00 2024-03-25T18:35:39.966 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 57 kB/s | 31 kB 00:00 2024-03-25T18:35:40.108 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T18:35:40.233 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-25T18:35:40.575 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-25T18:35:40.633 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-25T18:35:40.859 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.1 MB/s | 2.0 MB 00:00 2024-03-25T18:35:40.884 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-25T18:35:40.926 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-25T18:35:41.026 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-25T18:35:41.093 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 147 kB/s | 31 kB 00:00 2024-03-25T18:35:41.763 INFO:teuthology.orchestra.run.smithi076.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 411 kB/s | 885 kB 00:02 2024-03-25T18:35:41.765 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:41.765 INFO:teuthology.orchestra.run.smithi076.stdout:Total 447 kB/s | 964 kB 00:02 2024-03-25T18:35:41.765 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:41.789 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:41.789 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:41.868 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:41.868 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:42.047 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:42.208 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:35:42.281 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:42.994 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:35:42.994 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:35:43.312 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:35:43.312 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:43.312 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:43.312 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:35:43.312 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:43.312 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:43.609 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-volume 2024-03-25T18:35:44.190 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:44.341 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:44.342 INFO:teuthology.orchestra.run.smithi076.stdout:Install 2 Packages 2024-03-25T18:35:44.343 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:44.343 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 332 k 2024-03-25T18:35:44.343 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 1.5 M 2024-03-25T18:35:44.343 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:44.443 INFO:teuthology.orchestra.run.smithi076.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 692 kB/s | 69 kB 00:00 2024-03-25T18:35:45.076 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 322 kB/s | 1.8 MB 00:05 2024-03-25T18:35:45.083 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:45.083 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.2 MB/s | 6.9 MB 00:05 2024-03-25T18:35:45.102 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:35:45.123 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:35:45.123 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:35:45.235 INFO:teuthology.orchestra.run.smithi076.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 296 kB/s | 264 kB 00:00 2024-03-25T18:35:45.236 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:45.236 INFO:teuthology.orchestra.run.smithi076.stdout:Total 372 kB/s | 332 kB 00:00 2024-03-25T18:35:45.236 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:45.245 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:45.245 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:45.286 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:45.286 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:45.302 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:35:45.302 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:35:45.433 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:35:45.654 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:45.669 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:35:45.670 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:35:45.670 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:35:45.670 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:45.821 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:35:45.902 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:35:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:35:45.984 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:35:46.014 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:35:46.119 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:35:46.269 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:35:46.362 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:35:46.411 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:35:46.439 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:35:46.471 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:35:46.727 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:35:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:35:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:46.762 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:35:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:46.763 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:46.763 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:47.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados-devel 2024-03-25T18:35:47.570 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:35:47.570 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:35:47.570 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:35:47.570 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:35:47.570 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:35:47.571 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:35:47.679 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:47.828 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:47.829 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:35:47.830 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:47.830 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 127 k 2024-03-25T18:35:47.830 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 456 k 2024-03-25T18:35:47.830 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:47.920 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:35:47.920 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:47.921 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:35:48.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:35:48.723 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:35:48.881 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:35:48.882 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================================ 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-03-25T18:35:48.883 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:35:48.884 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 37 M 2024-03-25T18:35:48.884 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 209 M 2024-03-25T18:35:48.884 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:35:48.957 INFO:teuthology.orchestra.run.smithi076.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 112 kB/s | 127 kB 00:01 2024-03-25T18:35:48.958 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:48.958 INFO:teuthology.orchestra.run.smithi076.stdout:Total 112 kB/s | 127 kB 00:01 2024-03-25T18:35:48.958 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:48.964 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:48.964 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:49.023 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:49.024 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:49.178 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:49.183 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 267 kB/s | 32 kB 00:00 2024-03-25T18:35:49.242 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 503 kB/s | 17 kB 00:00 2024-03-25T18:35:49.289 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:49.292 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 682 kB/s | 34 kB 00:00 2024-03-25T18:35:49.475 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-25T18:35:49.617 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T18:35:49.668 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-03-25T18:35:49.707 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:49.982 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:49.982 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:49.982 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:49.982 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:49.982 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:49.982 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:50.001 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2024-03-25T18:35:50.202 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-25T18:35:50.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs2 2024-03-25T18:35:50.327 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-25T18:35:50.352 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-25T18:35:50.812 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:50.860 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 460 kB/s | 245 kB 00:00 2024-03-25T18:35:50.910 INFO:teuthology.orchestra.run.smithi076.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:50.966 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:50.967 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:50.967 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:51.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs-devel 2024-03-25T18:35:51.615 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:51.763 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:51.764 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:35:51.764 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:51.764 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 31 k 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 140 k 2024-03-25T18:35:51.765 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:52.294 INFO:teuthology.orchestra.run.smithi076.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 59 kB/s | 31 kB 00:00 2024-03-25T18:35:52.295 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:52.295 INFO:teuthology.orchestra.run.smithi076.stdout:Total 59 kB/s | 31 kB 00:00 2024-03-25T18:35:52.295 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:52.299 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:52.299 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:52.322 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:52.322 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:52.465 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:52.552 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:52.931 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:53.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:53.190 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:53.190 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:35:53.190 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:53.190 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:53.190 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:53.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados2 2024-03-25T18:35:53.918 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:54.014 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:54.066 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:54.067 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:54.067 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:54.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librbd1 2024-03-25T18:35:54.682 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:54.778 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:54.831 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:54.832 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:54.832 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:54.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rados 2024-03-25T18:35:55.442 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:55.537 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:55.589 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:55.590 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:55.590 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:55.639 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rgw 2024-03-25T18:35:56.209 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:50 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:56.310 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:56.364 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:56.366 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:56.366 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:56.428 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-cephfs 2024-03-25T18:35:57.023 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:57.122 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:57.178 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:57.179 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:57.179 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:57.232 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rbd 2024-03-25T18:35:57.798 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:57.895 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:35:57.946 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:57.947 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T18:35:57.947 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:35:58.003 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-fuse 2024-03-25T18:35:58.567 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:35:58.716 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:35:58.716 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:58.716 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-03-25T18:35:58.716 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:58.716 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:35:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:35:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:58.717 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:35:58.717 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:35:58.717 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:35:58.717 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:35:58.717 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 85 k 2024-03-25T18:35:58.718 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 231 k 2024-03-25T18:35:58.718 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:35:59.589 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 98 kB/s | 85 kB 00:00 2024-03-25T18:35:59.590 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:59.590 INFO:teuthology.orchestra.run.smithi076.stdout:Total 98 kB/s | 85 kB 00:00 2024-03-25T18:35:59.591 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:35:59.598 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:35:59.598 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:35:59.656 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:35:59.657 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:35:59.791 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:35:59.888 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:00.281 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:00.572 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:00.572 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:00.572 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:36:00.572 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:00.572 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:00.572 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:36:00.798 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-mirror 2024-03-25T18:36:01.374 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:36:01.526 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:36:01.530 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:01.531 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:36:01.531 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:01.531 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 3.1 M 2024-03-25T18:36:01.531 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 13 M 2024-03-25T18:36:01.531 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:36:11.434 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 319 kB/s | 3.1 MB 00:09 2024-03-25T18:36:11.435 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:11.435 INFO:teuthology.orchestra.run.smithi076.stdout:Total 319 kB/s | 3.1 MB 00:09 2024-03-25T18:36:11.435 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:36:11.446 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:36:11.446 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:36:11.523 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:36:11.524 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:36:11.960 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:36:12.018 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:12.043 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:12.043 INFO:teuthology.orchestra.run.smithi076.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:36:12.043 INFO:teuthology.orchestra.run.smithi076.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:36:12.043 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:12.043 INFO:teuthology.orchestra.run.smithi076.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:36:12.043 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:12.342 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 325 kB/s | 7.4 MB 00:23 2024-03-25T18:36:12.345 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:12.345 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.6 MB/s | 37 MB 00:23 2024-03-25T18:36:12.388 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:12.415 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:12.415 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:12.786 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:12.786 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:13.164 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:13.164 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:13.164 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:36:13.164 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:13.164 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:13.164 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:36:13.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-nbd 2024-03-25T18:36:13.504 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:13.590 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:36:13.621 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:36:13.712 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:36:13.749 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:36:13.934 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:36:13.965 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:36:13.991 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:36:14.053 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:36:14.143 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:36:14.143 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:14.143 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2024-03-25T18:36:14.143 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:14.143 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 171 k 2024-03-25T18:36:14.144 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 494 k 2024-03-25T18:36:14.145 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:36:14.741 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:36:14.857 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:36:15.796 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 104 kB/s | 171 kB 00:01 2024-03-25T18:36:15.796 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:15.797 INFO:teuthology.orchestra.run.smithi076.stdout:Total 104 kB/s | 171 kB 00:01 2024-03-25T18:36:15.797 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:36:15.806 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:36:15.806 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:36:15.866 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:36:15.867 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:36:16.018 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:36:16.135 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:16.164 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:36:16.563 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:16.823 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:16.823 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:16.823 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:36:16.823 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:16.823 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:16.823 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:36:17.076 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:36:17.255 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:36:17.281 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:36:17.645 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:36:17.798 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:17.799 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:36:17.800 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:36:17.800 INFO:teuthology.orchestra.run.smithi076.stdout:Install 2 Packages 2024-03-25T18:36:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:36:17.800 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 91 M 2024-03-25T18:36:17.800 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 498 M 2024-03-25T18:36:17.800 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:36:19.931 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:36:19.932 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:36:20.376 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:36:20.377 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:36:20.378 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:36:20.378 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:36:20.378 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:36:20.378 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:20.378 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:20.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:36:21.339 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:21.495 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:36:21.497 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:36:21.498 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2024-03-25T18:36:21.499 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:21.499 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.0 M 2024-03-25T18:36:21.499 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-03-25T18:36:21.499 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:36:21.675 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 187 kB/s | 24 kB 00:00 2024-03-25T18:36:21.725 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 100 kB/s | 18 kB 00:00 2024-03-25T18:36:21.776 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:36:21.801 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:36:21.826 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:36:21.851 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 870 kB/s | 43 kB 00:00 2024-03-25T18:36:22.026 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 86 kB/s | 15 kB 00:00 2024-03-25T18:36:22.052 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-25T18:36:22.235 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 76 kB/s | 52 kB 00:00 2024-03-25T18:36:22.327 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 674 kB/s | 203 kB 00:00 2024-03-25T18:36:22.469 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 633 kB/s | 90 kB 00:00 2024-03-25T18:36:22.511 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 215 kB/s | 59 kB 00:00 2024-03-25T18:36:23.011 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.1 MB/s | 1.0 MB 00:00 2024-03-25T18:36:23.013 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:23.013 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-03-25T18:36:23.014 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:23.030 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:23.030 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:23.133 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:23.133 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:23.662 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:36:23.905 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:36:23.970 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:36:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:36:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:36:24.152 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:36:24.261 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:36:24.316 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:36:25.076 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:36:25.120 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:36:25.154 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:36:25.204 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:36:25.232 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:36:26.026 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:36:26.026 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:36:26.026 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:36:26.026 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:36:26.027 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:36:26.390 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:36:26.390 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:26.390 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:26.390 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:36:26.390 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:26.391 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:26.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:36:27.182 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:27.338 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:27.339 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:36:27.339 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:36:27.339 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:36:27.339 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:27.339 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:36:27.339 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-03-25T18:36:27.340 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:27.341 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 6.8 M 2024-03-25T18:36:27.341 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2024-03-25T18:36:27.341 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:36:27.780 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T18:36:27.813 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 803 kB/s | 248 kB 00:00 2024-03-25T18:36:28.148 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-25T18:36:28.239 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 557 kB/s | 51 kB 00:00 2024-03-25T18:36:28.265 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:36:28.290 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 176 kB/s | 138 kB 00:00 2024-03-25T18:36:28.292 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:28.293 INFO:teuthology.orchestra.run.smithi050.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-03-25T18:36:28.305 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:28.316 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:28.316 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:28.407 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:28.408 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:28.720 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:28.896 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:36:28.942 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:36:29.489 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:36:29.579 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:36:29.641 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:36:29.670 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:36:30.474 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:36:30.780 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:36:30.780 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:30.780 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:30.780 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:36:30.780 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:36:30.781 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:36:30.781 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:36:30.781 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:36:30.781 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:36:30.781 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:30.781 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:31.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2024-03-25T18:36:31.579 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:31.736 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:31.736 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:31.736 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:36:31.736 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:31.736 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-03-25T18:36:31.737 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 964 k 2024-03-25T18:36:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.8 M 2024-03-25T18:36:31.738 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:36:31.866 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 611 kB/s | 79 kB 00:00 2024-03-25T18:36:34.725 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 296 kB/s | 885 kB 00:02 2024-03-25T18:36:34.726 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:34.727 INFO:teuthology.orchestra.run.smithi050.stdout:Total 322 kB/s | 964 kB 00:02 2024-03-25T18:36:34.727 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:34.751 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:34.751 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:34.830 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:34.831 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:35.017 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:35.155 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:36:35.255 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:36:35.908 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:36:35.908 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:36:36.221 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:36:36.221 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:36.221 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:36.221 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:36:36.221 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:36.222 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:36.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2024-03-25T18:36:37.051 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:27 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:37.207 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:37.208 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:37.209 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-03-25T18:36:37.209 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:37.209 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 332 k 2024-03-25T18:36:37.209 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.5 M 2024-03-25T18:36:37.209 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:36:37.321 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 618 kB/s | 69 kB 00:00 2024-03-25T18:36:38.113 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 292 kB/s | 264 kB 00:00 2024-03-25T18:36:38.114 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:38.115 INFO:teuthology.orchestra.run.smithi050.stdout:Total 367 kB/s | 332 kB 00:00 2024-03-25T18:36:38.115 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:38.125 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:38.125 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:38.163 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:38.163 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:38.503 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:36:38.532 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:36:38.547 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:36:38.548 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:36:38.548 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:36:38.548 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:39.265 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:36:39.587 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:36:39.587 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:39.587 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:39.587 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:36:39.587 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:36:39.587 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:39.587 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:39.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2024-03-25T18:36:40.466 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:40.624 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:40.624 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:36:40.625 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:40.626 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 127 k 2024-03-25T18:36:40.626 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2024-03-25T18:36:40.626 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:36:41.635 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 125 kB/s | 127 kB 00:01 2024-03-25T18:36:41.636 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:41.636 INFO:teuthology.orchestra.run.smithi050.stdout:Total 125 kB/s | 127 kB 00:01 2024-03-25T18:36:41.637 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:41.643 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:41.644 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:41.706 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:41.707 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:41.875 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:41.987 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:42.396 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:42.675 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:42.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:42.676 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:42.676 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:42.676 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:42.676 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:42.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2024-03-25T18:36:43.475 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:43.575 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:43.632 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:43.633 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:36:43.633 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:43.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2024-03-25T18:36:44.266 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:34 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:44.422 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:36:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:44.424 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 k 2024-03-25T18:36:44.424 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 140 k 2024-03-25T18:36:44.424 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:36:45.047 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 50 kB/s | 31 kB 00:00 2024-03-25T18:36:45.048 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:45.048 INFO:teuthology.orchestra.run.smithi050.stdout:Total 50 kB/s | 31 kB 00:00 2024-03-25T18:36:45.048 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:45.052 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:45.052 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:45.079 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:45.080 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:45.203 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:45.672 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:45.937 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:45.938 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:45.938 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:45.938 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:46.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2024-03-25T18:36:46.685 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:46.785 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:46.840 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:46.841 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:36:46.841 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:46.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2024-03-25T18:36:47.457 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:47.556 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:47.611 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:47.612 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:36:47.612 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:47.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2024-03-25T18:36:48.240 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:48.343 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:48.398 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:48.399 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:36:48.399 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:48.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2024-03-25T18:36:49.034 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:49.134 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:49.188 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:49.190 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:36:49.190 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:49.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2024-03-25T18:36:49.809 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:49.909 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:49.964 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:49.965 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:36:49.965 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:50.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2024-03-25T18:36:50.584 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:50.684 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:36:50.738 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:50.739 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T18:36:50.739 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:50.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2024-03-25T18:36:51.360 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:51.514 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:51.515 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:51.516 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:36:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:51.516 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 85 k 2024-03-25T18:36:51.516 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 231 k 2024-03-25T18:36:51.516 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:36:52.250 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 116 kB/s | 85 kB 00:00 2024-03-25T18:36:52.251 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:36:52.251 INFO:teuthology.orchestra.run.smithi050.stdout:Total 116 kB/s | 85 kB 00:00 2024-03-25T18:36:52.252 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:36:52.261 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:36:52.261 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:36:52.324 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:36:52.324 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:36:52.473 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:36:52.593 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:52.987 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:53.276 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:36:53.276 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:53.276 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:36:53.276 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:36:53.276 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:53.276 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:36:53.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2024-03-25T18:36:54.082 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:36:54.238 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:36:54.239 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:54.239 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.1 M 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 13 M 2024-03-25T18:36:54.240 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:37:00.478 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 507 kB/s | 3.1 MB 00:06 2024-03-25T18:37:00.478 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:37:00.479 INFO:teuthology.orchestra.run.smithi050.stdout:Total 507 kB/s | 3.1 MB 00:06 2024-03-25T18:37:00.479 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:37:00.491 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:37:00.491 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:37:00.572 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:37:00.572 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:37:00.962 INFO:teuthology.orchestra.run.smithi076.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 533 kB/s | 22 MB 00:43 2024-03-25T18:37:01.027 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:37:01.087 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:37:01.112 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:37:01.112 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:37:01.112 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:37:01.112 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:37:01.112 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:37:01.112 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:02.275 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:37:02.275 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:02.275 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:37:02.275 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:02.275 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:02.276 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:37:02.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2024-03-25T18:37:03.110 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:37:03.265 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:37:03.266 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:37:03.267 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:03.267 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 171 k 2024-03-25T18:37:03.267 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 494 k 2024-03-25T18:37:03.267 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:37:04.189 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 186 kB/s | 171 kB 00:00 2024-03-25T18:37:04.190 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:37:04.190 INFO:teuthology.orchestra.run.smithi050.stdout:Total 186 kB/s | 171 kB 00:00 2024-03-25T18:37:04.191 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:37:04.201 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:37:04.201 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:37:04.265 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:37:04.265 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:37:04.422 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:37:04.525 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:37:04.979 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:37:05.267 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:37:05.267 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:05.267 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:37:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:05.268 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:05.268 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:37:05.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:37:06.055 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:37:06.212 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:37:06.213 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:37:06.214 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-03-25T18:37:06.214 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:37:06.214 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 91 M 2024-03-25T18:37:06.214 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 498 M 2024-03-25T18:37:06.214 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:38:00.096 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 427 kB/s | 22 MB 00:53 2024-03-25T18:38:43.392 INFO:teuthology.orchestra.run.smithi076.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 479 kB/s | 68 MB 02:25 2024-03-25T18:38:43.393 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:43.394 INFO:teuthology.orchestra.run.smithi076.stdout:Total 637 kB/s | 91 MB 02:25 2024-03-25T18:38:43.394 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:38:43.398 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:38:43.398 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:38:43.985 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:38:43.985 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:38:47.088 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:38:54.205 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:38:54.382 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:55.598 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 638 kB/s | 68 MB 01:49 2024-03-25T18:38:55.600 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:55.600 INFO:teuthology.orchestra.run.smithi050.stdout:Total 848 kB/s | 91 MB 01:49 2024-03-25T18:38:55.600 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:38:55.604 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:38:55.604 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:38:56.215 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:38:56.215 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:38:57.891 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:38:57.892 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:38:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:38:58.158 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:38:58.158 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:38:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:58.159 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:38:58.159 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:38:58.428 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:38:58.986 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:07:52 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:38:59.134 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:38:59.135 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:38:59.136 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 416 M 2024-03-25T18:38:59.136 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 1.7 G 2024-03-25T18:38:59.136 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:38:59.504 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:39:07.197 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:39:07.267 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:10.753 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:39:10.753 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:39:11.025 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:39:11.025 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:39:11.025 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:39:11.025 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:11.026 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:39:11.026 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:39:11.026 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:39:11.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:39:11.798 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:08:01 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:39:11.957 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:39:11.957 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:39:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:39:11.957 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:39:11.957 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:39:11.957 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:39:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:39:11.958 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:39:11.958 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:39:11.958 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:39:11.958 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:39:11.958 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 416 M 2024-03-25T18:39:11.958 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.7 G 2024-03-25T18:39:11.959 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:56:07.641 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 419 kB/s | 416 MB 16:55 2024-03-25T18:56:07.641 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:07.641 INFO:teuthology.orchestra.run.smithi050.stdout:Total 419 kB/s | 416 MB 16:55 2024-03-25T18:56:07.642 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:56:07.645 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:56:07.645 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:56:08.383 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 414 kB/s | 416 MB 17:09 2024-03-25T18:56:08.384 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:08.384 INFO:teuthology.orchestra.run.smithi076.stdout:Total 414 kB/s | 416 MB 17:09 2024-03-25T18:56:08.384 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:56:08.388 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:56:08.389 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:56:10.232 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:56:10.232 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:56:10.851 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:56:10.852 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:56:53.067 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:56:53.178 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:56:54.985 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:56:55.086 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:00.367 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:00.368 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:00.368 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:57:00.368 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:00.368 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:00.368 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:57:00.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:57:01.180 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:25:51 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:57:01.339 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:57:01.340 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:57:01.340 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:57:01.340 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:57:01.340 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:57:01.340 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:57:01.340 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:01.341 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:57:01.341 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================== 2024-03-25T18:57:01.341 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:57:01.341 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:01.341 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.4 M 2024-03-25T18:57:01.341 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 6.4 M 2024-03-25T18:57:01.341 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:57:02.689 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:02.931 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:57:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:02.931 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:57:03.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:57:03.770 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:25:57 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:57:03.921 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:57:03.922 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:57:03.922 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:57:03.922 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:57:03.922 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:57:03.922 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:57:03.922 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:03.922 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:57:03.923 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2024-03-25T18:57:03.923 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:57:03.923 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:03.923 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 1.4 M 2024-03-25T18:57:03.923 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 6.4 M 2024-03-25T18:57:03.923 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T18:57:07.705 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 230 kB/s | 1.4 MB 00:06 2024-03-25T18:57:07.706 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:57:07.706 INFO:teuthology.orchestra.run.smithi050.stdout:Total 230 kB/s | 1.4 MB 00:06 2024-03-25T18:57:07.707 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T18:57:07.711 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T18:57:07.712 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T18:57:07.758 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T18:57:07.759 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T18:57:08.025 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T18:57:08.122 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:08.533 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:08.819 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:08.820 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T18:57:08.820 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:08.820 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:08.820 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T18:57:09.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:57:09.599 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:25:59 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T18:57:09.752 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T18:57:09.753 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:57:09.753 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T18:57:09.753 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 176 M 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 753 M 2024-03-25T18:57:09.754 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T18:57:09.799 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 249 kB/s | 1.4 MB 00:05 2024-03-25T18:57:09.799 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:57:09.800 INFO:teuthology.orchestra.run.smithi076.stdout:Total 249 kB/s | 1.4 MB 00:05 2024-03-25T18:57:09.800 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T18:57:09.804 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T18:57:09.804 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T18:57:09.850 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T18:57:09.850 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T18:57:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T18:57:10.206 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:10.910 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T18:57:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:10.910 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:10.910 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T18:57:11.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:57:11.661 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:26:05 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T18:57:11.809 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T18:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T18:57:11.810 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 176 M 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 753 M 2024-03-25T18:57:11.811 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:04:24.045 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 414 kB/s | 176 MB 07:14 2024-03-25T19:04:24.046 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:24.046 INFO:teuthology.orchestra.run.smithi050.stdout:Total 414 kB/s | 176 MB 07:14 2024-03-25T19:04:24.046 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:04:24.048 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:04:24.048 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:04:25.159 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:04:25.159 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:04:44.241 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:04:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:04:49.093 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:04:49.361 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:04:49.361 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:04:49.361 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:04:49.362 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:04:49.362 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:04:49.362 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:04:49.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:04:50.119 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:33:40 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:04:50.273 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:04:50.274 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:04:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:04:50.274 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:04:50.274 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:04:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:04:50.274 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:04:50.275 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:04:50.275 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:04:50.275 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:04:50.275 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:04:50.275 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 672 M 2024-03-25T19:04:50.275 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 3.1 G 2024-03-25T19:04:50.275 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:05:27.739 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 363 kB/s | 176 MB 08:15 2024-03-25T19:05:27.740 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:05:27.740 INFO:teuthology.orchestra.run.smithi076.stdout:Total 363 kB/s | 176 MB 08:15 2024-03-25T19:05:27.740 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:05:27.744 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:05:27.744 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:05:28.806 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:05:28.806 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:05:46.786 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:05:46.961 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:51.761 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:05:52.027 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:05:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:05:52.027 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:05:52.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:05:52.804 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:34:46 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:05:52.952 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:05:52.953 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:05:52.953 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:05:52.953 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:05:52.953 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:05:52.953 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:05:52.953 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:05:52.953 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:05:52.954 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:05:52.954 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:05:52.954 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:05:52.954 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 672 M 2024-03-25T19:05:52.954 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 3.1 G 2024-03-25T19:05:52.954 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:25:28.407 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 556 kB/s | 672 MB 20:38 2024-03-25T19:25:28.407 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:25:28.407 INFO:teuthology.orchestra.run.smithi050.stdout:Total 556 kB/s | 672 MB 20:38 2024-03-25T19:25:28.407 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:25:28.412 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:25:28.412 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:25:32.580 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:25:32.581 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:26:47.751 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:26:47.977 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:57.270 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:26:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:26:57.611 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:26:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:57.611 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:26:57.611 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:26:57.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:26:58.457 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:55:48 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:26:58.556 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:26:58.611 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:26:58.612 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-03-25T19:26:58.612 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:26:58.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:26:59.247 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:55:49 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:26:59.402 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:26:59.403 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:26:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:26:59.403 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:26:59.403 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:26:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:26:59.403 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:26:59.404 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:26:59.404 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:26:59.404 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:26:59.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:26:59.404 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 20 M 2024-03-25T19:26:59.404 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 103 M 2024-03-25T19:26:59.404 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:27:18.568 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 20 MB 00:19 2024-03-25T19:27:18.569 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:18.569 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.0 MB/s | 20 MB 00:19 2024-03-25T19:27:18.569 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:27:18.573 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:27:18.574 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:27:18.731 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:27:18.731 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:27:21.088 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:27:21.186 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:22.201 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:27:22.483 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:27:22.669 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:27:23.238 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:56:13 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:27:23.393 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:27:23.393 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 48 M 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 173 M 2024-03-25T19:27:23.394 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:28:27.498 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 762 kB/s | 48 MB 01:04 2024-03-25T19:28:27.499 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:28:27.499 INFO:teuthology.orchestra.run.smithi050.stdout:Total 762 kB/s | 48 MB 01:04 2024-03-25T19:28:27.499 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:28:27.502 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:28:27.830 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:28:27.831 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:28:32.853 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:28:32.948 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:28:34.311 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:28:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:28:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:28:34.589 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:28:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:28:34.589 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:28:34.589 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:28:34.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:28:35.370 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:57:25 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:28:35.525 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:28:35.526 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:28:35.526 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:28:35.526 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:28:35.526 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 99 M 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 449 M 2024-03-25T19:28:35.527 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:29:49.168 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 99 MB 01:13 2024-03-25T19:29:49.168 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:49.168 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.3 MB/s | 99 MB 01:13 2024-03-25T19:29:49.168 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:29:49.171 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:29:49.171 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:29:49.811 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:29:49.811 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:30:00.705 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:30:00.800 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:03.843 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:04.118 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:30:04.118 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:30:04.118 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:30:04.118 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:30:04.118 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:30:04.119 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:30:04.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:30:04.878 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:58:54 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:30:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:30:05.032 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:30:05.033 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:30:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 287 M 2024-03-25T19:30:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.2 G 2024-03-25T19:30:05.034 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:30:50.957 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 460 kB/s | 672 MB 24:58 2024-03-25T19:30:50.958 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:50.959 INFO:teuthology.orchestra.run.smithi076.stdout:Total 460 kB/s | 672 MB 24:58 2024-03-25T19:30:50.959 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:30:50.963 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:30:50.963 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:30:54.931 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:30:54.931 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:32:05.686 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:32:06.290 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:15.151 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:15.425 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:32:15.425 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:15.425 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:32:15.425 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:15.425 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:15.425 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:32:15.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:32:16.304 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:01:10 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:32:16.399 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:32:16.451 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:32:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2024-03-25T19:32:16.452 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:32:16.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:32:17.106 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:01:11 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:32:17.254 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:32:17.255 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:32:17.256 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:32:17.256 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:17.256 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 20 M 2024-03-25T19:32:17.256 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 103 M 2024-03-25T19:32:17.256 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:32:38.585 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 946 kB/s | 20 MB 00:21 2024-03-25T19:32:38.586 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:38.586 INFO:teuthology.orchestra.run.smithi076.stdout:Total 946 kB/s | 20 MB 00:21 2024-03-25T19:32:38.586 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:32:38.590 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:32:38.590 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:32:38.740 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:32:38.740 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:32:40.962 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:32:41.133 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:42.176 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:42.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:42.445 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:42.446 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:32:42.446 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:42.446 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:42.446 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:32:42.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:32:43.210 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:01:37 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:32:43.359 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:32:43.360 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:32:43.361 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 48 M 2024-03-25T19:32:43.361 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2024-03-25T19:32:43.361 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:33:19.967 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 48 MB 00:36 2024-03-25T19:33:19.968 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:33:19.969 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.3 MB/s | 48 MB 00:36 2024-03-25T19:33:19.969 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:33:19.972 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:33:19.972 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:33:20.284 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:33:20.284 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:33:25.001 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:33:25.173 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:26.650 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:26.898 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:33:26.899 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:33:26.899 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:33:26.899 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:33:26.899 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:33:26.899 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:33:27.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:33:27.669 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:02:21 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:33:27.817 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:33:27.818 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:33:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:33:27.818 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:33:27.818 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:33:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:33:27.818 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:33:27.819 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:33:27.819 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:33:27.819 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:33:27.819 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:33:27.819 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 99 M 2024-03-25T19:33:27.819 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 449 M 2024-03-25T19:33:27.819 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:35:07.061 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 99 MB 01:39 2024-03-25T19:35:07.061 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:07.062 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.0 MB/s | 99 MB 01:39 2024-03-25T19:35:07.062 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:35:07.065 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:35:07.065 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:35:07.672 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:35:07.672 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:35:11.710 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 958 kB/s | 287 MB 05:06 2024-03-25T19:35:11.711 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:11.711 INFO:teuthology.orchestra.run.smithi050.stdout:Total 958 kB/s | 287 MB 05:06 2024-03-25T19:35:11.711 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:35:11.715 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:35:11.715 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:35:13.504 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:35:13.505 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:35:17.930 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:35:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:21.257 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:21.257 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:35:21.257 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:35:21.257 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:21.257 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:35:21.257 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:35:21.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:35:22.032 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:04:16 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:35:22.179 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:35:22.180 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:35:22.181 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 287 M 2024-03-25T19:35:22.181 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 1.2 G 2024-03-25T19:35:22.181 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:35:44.613 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:35:44.718 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:45.956 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:46.255 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:46.256 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:35:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:46.256 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:46.256 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:35:46.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:35:47.047 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:04:37 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:35:47.204 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:35:47.205 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:47.206 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 5.9 M 2024-03-25T19:35:47.206 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 28 M 2024-03-25T19:35:47.206 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:35:50.094 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 2.1 MB/s | 5.9 MB 00:02 2024-03-25T19:35:50.095 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:50.095 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.1 MB/s | 5.9 MB 00:02 2024-03-25T19:35:50.096 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:35:50.100 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:35:50.100 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:35:50.174 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:35:50.175 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:35:50.945 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:35:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:51.730 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:52.004 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:52.004 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:52.004 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:35:52.004 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:52.005 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:52.005 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:35:52.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:35:52.785 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:04:42 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:35:52.943 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:52.944 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:35:52.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:52.945 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 109 k 2024-03-25T19:35:52.945 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 440 k 2024-03-25T19:35:52.945 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:35:53.522 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 188 kB/s | 109 kB 00:00 2024-03-25T19:35:53.523 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:53.523 INFO:teuthology.orchestra.run.smithi050.stdout:Total 188 kB/s | 109 kB 00:00 2024-03-25T19:35:53.523 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:35:53.526 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:35:53.526 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:35:53.564 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:35:53.564 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:35:53.680 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:35:53.783 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:54.149 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:54.431 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:54.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:54.431 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:35:54.431 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:54.431 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:54.431 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:35:54.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:35:55.233 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:04:45 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:35:55.391 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:35:55.391 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 5.4 M 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 25 M 2024-03-25T19:35:55.392 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:35:58.833 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.6 MB/s | 5.4 MB 00:03 2024-03-25T19:35:58.834 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:58.834 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.6 MB/s | 5.4 MB 00:03 2024-03-25T19:35:58.835 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:35:58.839 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:35:58.839 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:35:58.910 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:35:58.910 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:35:59.609 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:35:59.710 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:00.215 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:00.506 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:36:00.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:00.506 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:36:00.506 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:00.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:00.506 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:36:00.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2-debuginfo 2024-03-25T19:36:01.286 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:04:51 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:36:01.440 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:36:01.441 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:36:01.441 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:36:01.441 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:36:01.441 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:36:01.441 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:36:01.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:01.441 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:36:01.442 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:36:01.442 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:36:01.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:01.442 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 44 M 2024-03-25T19:36:01.442 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 253 M 2024-03-25T19:36:01.442 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:36:21.836 INFO:teuthology.orchestra.run.smithi050.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 2.2 MB/s | 44 MB 00:20 2024-03-25T19:36:21.837 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:21.837 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.2 MB/s | 44 MB 00:20 2024-03-25T19:36:21.837 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:36:21.842 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:36:21.842 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:36:22.155 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:36:22.155 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:36:27.494 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:36:27.622 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:29.458 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:29.767 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:29.768 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:36:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:29.768 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:29.768 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:36:29.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:36:30.563 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:05:20 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:36:30.717 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:36:30.717 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:36:30.718 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:36:30.719 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 52 M 2024-03-25T19:36:30.719 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 308 M 2024-03-25T19:36:30.719 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:36:59.228 INFO:teuthology.orchestra.run.smithi050.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.8 MB/s | 52 MB 00:28 2024-03-25T19:36:59.229 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:59.229 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.8 MB/s | 52 MB 00:28 2024-03-25T19:36:59.229 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:36:59.233 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:36:59.234 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:36:59.595 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:36:59.595 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:37:05.943 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:37:06.054 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:37:08.282 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:37:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:37:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:08.553 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:37:08.553 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:08.554 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:08.554 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:37:08.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:37:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:05:59 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:37:09.505 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:37:09.505 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 615 k 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.1 M 2024-03-25T19:37:09.506 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:37:10.540 INFO:teuthology.orchestra.run.smithi050.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 595 kB/s | 615 kB 00:01 2024-03-25T19:37:10.540 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:10.541 INFO:teuthology.orchestra.run.smithi050.stdout:Total 594 kB/s | 615 kB 00:01 2024-03-25T19:37:10.541 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:37:10.545 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:37:10.545 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:37:10.588 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:37:10.589 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:37:10.789 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:37:10.886 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:11.300 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:11.569 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:11.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:11.570 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:37:11.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:11.570 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:11.570 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:37:11.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:37:12.366 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:06:02 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:37:12.524 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:37:12.525 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:12.525 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:37:12.525 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:12.525 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:37:12.525 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:37:12.525 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:12.525 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:37:12.526 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:12.526 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:37:12.526 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:12.526 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 771 k 2024-03-25T19:37:12.526 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.7 M 2024-03-25T19:37:12.526 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:37:13.626 INFO:teuthology.orchestra.run.smithi050.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 701 kB/s | 771 kB 00:01 2024-03-25T19:37:13.627 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:13.627 INFO:teuthology.orchestra.run.smithi050.stdout:Total 700 kB/s | 771 kB 00:01 2024-03-25T19:37:13.627 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:37:13.631 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:37:13.631 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:37:13.675 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:37:13.675 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:37:13.866 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:37:13.987 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:37:14.379 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:37:14.661 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:37:14.661 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:14.661 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:37:14.661 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:14.661 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:14.661 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:37:14.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:37:15.548 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:06:05 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:37:15.702 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:15.703 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.1 M 2024-03-25T19:37:15.704 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 4.3 M 2024-03-25T19:37:15.704 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:37:16.955 INFO:teuthology.orchestra.run.smithi050.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 920 kB/s | 1.1 MB 00:01 2024-03-25T19:37:16.956 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:16.956 INFO:teuthology.orchestra.run.smithi050.stdout:Total 919 kB/s | 1.1 MB 00:01 2024-03-25T19:37:16.956 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:37:16.960 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:37:16.960 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:37:17.006 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:37:17.006 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:37:17.239 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:37:17.356 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:17.763 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:18.060 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:37:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:18.060 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:37:18.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:37:18.850 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:06:08 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:37:19.005 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:37:19.006 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:19.006 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:37:19.006 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:19.006 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:37:19.006 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:37:19.006 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:19.007 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:37:19.007 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:19.007 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:37:19.007 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:19.007 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 337 k 2024-03-25T19:37:19.007 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.0 M 2024-03-25T19:37:19.007 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:37:19.833 INFO:teuthology.orchestra.run.smithi050.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 408 kB/s | 337 kB 00:00 2024-03-25T19:37:19.834 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:19.834 INFO:teuthology.orchestra.run.smithi050.stdout:Total 408 kB/s | 337 kB 00:00 2024-03-25T19:37:19.834 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:37:19.838 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:37:19.838 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:37:19.877 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:37:19.878 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:37:20.022 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:37:20.135 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:20.529 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:20.810 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:20.810 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:20.810 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:37:20.810 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:20.810 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:20.810 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:37:21.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:37:21.604 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:06:11 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:37:21.760 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:37:21.760 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 509 k 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.5 M 2024-03-25T19:37:21.761 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:37:23.878 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 241 kB/s | 509 kB 00:02 2024-03-25T19:37:23.879 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:23.879 INFO:teuthology.orchestra.run.smithi050.stdout:Total 240 kB/s | 509 kB 00:02 2024-03-25T19:37:23.879 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:37:23.883 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:37:23.883 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:37:23.924 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:37:23.925 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:37:24.091 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:37:24.205 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:24.586 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:24.859 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:24.859 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:24.859 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:37:24.859 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:24.859 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:24.859 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:37:25.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:37:25.632 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:06:15 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:37:25.788 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:37:25.788 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:25.788 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:37:25.788 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:25.788 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:37:25.789 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:37:25.789 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:25.789 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:37:25.789 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:37:25.789 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:37:25.789 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:37:25.789 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 52 M 2024-03-25T19:37:25.790 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 298 M 2024-03-25T19:37:25.790 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:37:54.970 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.8 MB/s | 52 MB 00:29 2024-03-25T19:37:54.971 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:54.972 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.8 MB/s | 52 MB 00:29 2024-03-25T19:37:54.972 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:37:54.976 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:37:54.976 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:37:55.329 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:37:55.329 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:37:56.228 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.9 MB/s | 287 MB 02:34 2024-03-25T19:37:56.229 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:56.229 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.9 MB/s | 287 MB 02:34 2024-03-25T19:37:56.229 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:37:56.233 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:37:56.233 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:37:57.937 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:37:57.937 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:38:01.700 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:38:01.806 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:03.929 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:38:04.203 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:38:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:04.203 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:38:04.203 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:38:04.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:38:04.986 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 1:06:54 ago on Mon 25 Mar 2024 06:31:10 PM UTC. 2024-03-25T19:38:05.144 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-03-25T19:38:05.144 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 1.3 M 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 5.5 M 2024-03-25T19:38:05.145 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-03-25T19:38:07.247 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 610 kB/s | 1.3 MB 00:02 2024-03-25T19:38:07.248 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:07.249 INFO:teuthology.orchestra.run.smithi050.stdout:Total 610 kB/s | 1.3 MB 00:02 2024-03-25T19:38:07.249 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-03-25T19:38:07.253 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-03-25T19:38:07.253 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-03-25T19:38:07.300 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-03-25T19:38:07.300 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-03-25T19:38:07.689 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-03-25T19:38:07.800 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:08.233 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:38:08.620 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-03-25T19:38:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:08.620 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:38:08.620 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-03-25T19:38:08.823 DEBUG:teuthology.parallel:result is None 2024-03-25T19:38:27.116 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:38:27.223 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:34.994 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:38:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:34.994 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:38:35.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:38:35.801 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:07:29 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:38:35.951 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:35.952 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:38:35.952 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:35.952 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 5.9 M 2024-03-25T19:38:35.952 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 28 M 2024-03-25T19:38:35.952 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:38:41.628 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.0 MB/s | 5.9 MB 00:05 2024-03-25T19:38:41.629 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:41.629 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.0 MB/s | 5.9 MB 00:05 2024-03-25T19:38:41.630 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:38:41.633 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:38:41.633 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:38:41.703 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:38:41.704 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:38:42.445 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:38:42.560 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:43.165 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:38:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:43.422 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:38:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:43.422 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:43.422 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:38:43.658 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:38:44.221 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:07:38 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:38:44.371 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:38:44.372 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:44.373 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:38:44.373 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:44.373 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 109 k 2024-03-25T19:38:44.373 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 440 k 2024-03-25T19:38:44.373 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:38:44.998 INFO:teuthology.orchestra.run.smithi076.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 174 kB/s | 109 kB 00:00 2024-03-25T19:38:44.998 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:44.999 INFO:teuthology.orchestra.run.smithi076.stdout:Total 174 kB/s | 109 kB 00:00 2024-03-25T19:38:44.999 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:38:45.002 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:38:45.002 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:38:45.036 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:38:45.036 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:38:45.157 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:38:45.287 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:45.682 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:45.948 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:38:45.949 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:45.949 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:38:45.949 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:45.949 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:45.949 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:38:46.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:38:46.731 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:07:40 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:38:46.879 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:38:46.879 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:38:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:46.881 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 5.4 M 2024-03-25T19:38:46.881 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 25 M 2024-03-25T19:38:46.881 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:38:49.959 INFO:teuthology.orchestra.run.smithi076.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 5.4 MB 00:03 2024-03-25T19:38:49.960 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:49.960 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.7 MB/s | 5.4 MB 00:03 2024-03-25T19:38:49.961 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:38:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:38:49.964 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:38:50.033 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:38:50.033 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:38:50.692 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:38:50.808 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:51.320 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:51.588 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:51.589 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:51.589 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:38:51.589 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:51.589 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:51.589 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:38:51.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados2-debuginfo 2024-03-25T19:38:52.390 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:07:46 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:38:52.539 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 44 M 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 253 M 2024-03-25T19:38:52.540 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:02.220 INFO:teuthology.orchestra.run.smithi076.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 4.6 MB/s | 44 MB 00:09 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:02.221 INFO:teuthology.orchestra.run.smithi076.stdout:Total 4.6 MB/s | 44 MB 00:09 2024-03-25T19:39:02.222 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:02.225 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:02.225 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:02.524 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:02.524 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:07.563 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:07.660 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:09.588 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:09.844 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:09.845 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:39:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:09.845 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:09.845 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:39:10.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:39:10.648 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:04 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:39:10.795 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:39:10.796 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:39:10.796 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:10.796 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:39:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:10.796 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:39:10.797 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:10.797 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:39:10.797 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:10.797 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 52 M 2024-03-25T19:39:10.797 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 308 M 2024-03-25T19:39:10.797 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:27.263 INFO:teuthology.orchestra.run.smithi076.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 3.1 MB/s | 52 MB 00:16 2024-03-25T19:39:27.264 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:27.264 INFO:teuthology.orchestra.run.smithi076.stdout:Total 3.1 MB/s | 52 MB 00:16 2024-03-25T19:39:27.264 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:27.268 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:27.268 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:27.613 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:27.613 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:33.588 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:33.878 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:36.131 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:39:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:36.389 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:39:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:36.389 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:39:36.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:39:37.187 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:31 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:37.336 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:39:37.337 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:37.337 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:39:37.337 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:37.337 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 615 k 2024-03-25T19:39:37.337 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 2.1 M 2024-03-25T19:39:37.337 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:38.041 INFO:teuthology.orchestra.run.smithi076.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 874 kB/s | 615 kB 00:00 2024-03-25T19:39:38.041 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:38.042 INFO:teuthology.orchestra.run.smithi076.stdout:Total 872 kB/s | 615 kB 00:00 2024-03-25T19:39:38.042 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:38.045 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:38.046 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:38.084 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:38.084 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:38.251 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:38.365 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:38.765 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:39.024 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:39:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:39.024 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:39:39.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:39:39.841 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:33 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:39:39.993 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:39.995 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 771 k 2024-03-25T19:39:39.995 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 2.7 M 2024-03-25T19:39:39.995 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:40.311 INFO:teuthology.orchestra.run.smithi076.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 2.4 MB/s | 771 kB 00:00 2024-03-25T19:39:40.312 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:40.312 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.4 MB/s | 771 kB 00:00 2024-03-25T19:39:40.312 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:40.315 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:40.315 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:40.353 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:40.354 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:40.530 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:40.627 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:41.040 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:39:41.306 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:41.307 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:39:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:41.307 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:41.307 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:39:41.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:39:42.125 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:36 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:39:42.273 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:39:42.274 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:42.274 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:39:42.274 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:42.274 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:39:42.274 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:39:42.274 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:42.275 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:39:42.275 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:42.275 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:39:42.275 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:42.275 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 1.1 M 2024-03-25T19:39:42.275 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 4.3 M 2024-03-25T19:39:42.275 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:43.242 INFO:teuthology.orchestra.run.smithi076.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 1.2 MB/s | 1.1 MB 00:00 2024-03-25T19:39:43.242 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:43.243 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.2 MB/s | 1.1 MB 00:00 2024-03-25T19:39:43.243 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:43.247 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:43.247 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:43.291 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:43.291 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:43.515 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:43.623 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:44.007 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:44.298 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:44.298 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:44.298 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:39:44.298 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:44.298 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:44.298 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:39:44.545 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:39:45.104 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:39 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:39:45.253 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:39:45.254 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:45.254 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 337 k 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 1.0 M 2024-03-25T19:39:45.255 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 542 kB/s | 337 kB 00:00 2024-03-25T19:39:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:45.878 INFO:teuthology.orchestra.run.smithi076.stdout:Total 541 kB/s | 337 kB 00:00 2024-03-25T19:39:45.879 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:45.882 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:45.882 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:45.920 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:45.921 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:46.056 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:46.168 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:46.536 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:39:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:46.807 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:39:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:46.807 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:39:47.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:39:47.635 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:41 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:39:47.783 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:39:47.784 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:47.785 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 509 k 2024-03-25T19:39:47.785 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 2.5 M 2024-03-25T19:39:47.785 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:48.214 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 509 kB 00:00 2024-03-25T19:39:48.215 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:48.215 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.2 MB/s | 509 kB 00:00 2024-03-25T19:39:48.215 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:48.218 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:48.218 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:48.255 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:48.256 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:48.420 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:48.530 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:48.922 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:49.190 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:49.190 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:49.190 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:39:49.190 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:49.190 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:49.191 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:39:49.436 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:39:49.996 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:43 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:39:50.148 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:39:50.149 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:39:50.150 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:39:50.150 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:39:50.150 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 52 M 2024-03-25T19:39:50.150 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 298 M 2024-03-25T19:39:50.150 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:39:52.642 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 21 MB/s | 52 MB 00:02 2024-03-25T19:39:52.642 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:52.643 INFO:teuthology.orchestra.run.smithi076.stdout:Total 21 MB/s | 52 MB 00:02 2024-03-25T19:39:52.643 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:39:52.647 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:39:52.647 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:39:52.981 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:39:52.982 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:39:58.872 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:39:59.140 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:01.425 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:01.695 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:01.695 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:01.695 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:40:01.696 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:01.696 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:01.696 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:40:01.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:40:02.506 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 1:08:56 ago on Mon 25 Mar 2024 06:31:06 PM UTC. 2024-03-25T19:40:02.656 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2024-03-25T19:40:02.657 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2024-03-25T19:40:02.658 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:02.658 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 1.3 M 2024-03-25T19:40:02.658 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 5.5 M 2024-03-25T19:40:02.658 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2024-03-25T19:40:03.852 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:40:03.852 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:03.853 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.0 MB/s | 1.3 MB 00:01 2024-03-25T19:40:03.853 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2024-03-25T19:40:03.856 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2024-03-25T19:40:03.856 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2024-03-25T19:40:03.900 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2024-03-25T19:40:03.901 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2024-03-25T19:40:04.164 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2024-03-25T19:40:04.267 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:04.727 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:04.995 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:04.995 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:04.995 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2024-03-25T19:40:04.995 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:04.995 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:04.995 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2024-03-25T19:40:05.249 DEBUG:teuthology.parallel:result is None 2024-03-25T19:40:05.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:40:05.509 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:40:05.533 INFO:teuthology.orchestra.run.smithi050.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:05.534 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:05.534 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:40:05.536 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:40:05.752 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:40:05.773 INFO:teuthology.orchestra.run.smithi076.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:05.774 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:05.774 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:40:05.777 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:40:05.777 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:05.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:40:05.810 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:05.810 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:40:05.842 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:40:05.842 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:05.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:40:05.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:40:05.949 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:05.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:40:05.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:40:06.044 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:40:06.044 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:06.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:40:06.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:40:06.142 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:06.143 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:40:06.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:40:06.236 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:40:06.236 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:06.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:40:06.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:40:06.332 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:06.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:40:06.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:40:06.425 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:40:06.503 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:40:06.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2024-03-25T19:40:06.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /var/log/ceph 2024-03-25T19:40:06.532 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:40:06.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:40:06.574 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:40:06.601 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:40:06.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:40:06.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:40:06.679 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:40:06.679 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:40:06.679 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621325', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9CW1XN++EKPZEb9HmxbLYSDkky3NYmzMESg0BgqpifYbRz3QKNUxTHlGiSxUPIXQ8WKSkuXfLz3ENkao/ZWKU=', 'smithi076.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD61+DYpDp+6uYpwZwXyqjynu8sPDA81HZl0StzJkQ3mELz3Zi7fNYKjpR9ksHYe8nHoeAss37muvbvByDQjCbg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:40:06.680 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:40:06.711 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:40:06.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:40:06.766 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:40:06.793 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:06.793 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:40:06.846 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:40:06.847 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-03-25T19:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:40:06.900 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-03-25 18:37:01.367206667 +0000 2024-03-25T19:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-03-25 18:30:14.539793768 +0000 2024-03-25T19:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-03-25 18:30:14.539793768 +0000 2024-03-25T19:40:06.901 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-03-25 18:30:14.539793768 +0000 2024-03-25T19:40:06.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:40:06.966 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-03-25T19:40:06.966 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-03-25T19:40:06.967 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.2129e-05 s, 5.6 MB/s 2024-03-25T19:40:06.968 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:40:07.023 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-03-25 18:37:01.367206667 +0000 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-03-25 18:30:14.877785490 +0000 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-03-25 18:30:14.877785490 +0000 2024-03-25T19:40:07.079 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-03-25 18:30:14.877785490 +0000 2024-03-25T19:40:07.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:40:07.143 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-03-25T19:40:07.143 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-03-25T19:40:07.143 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.7406e-05 s, 5.3 MB/s 2024-03-25T19:40:07.145 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:40:07.200 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-03-25T19:40:07.254 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-03-25 18:37:01.368206642 +0000 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-03-25 18:30:15.198777629 +0000 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-03-25 18:30:15.198777629 +0000 2024-03-25T19:40:07.255 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-03-25 18:30:15.198777629 +0000 2024-03-25T19:40:07.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:40:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-03-25T19:40:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-03-25T19:40:07.320 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 7.6391e-05 s, 6.7 MB/s 2024-03-25T19:40:07.321 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:40:07.376 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-03-25T19:40:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:40:07.430 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:07.430 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:40:07.430 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:07.430 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:07.431 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-03-25 18:37:01.368206642 +0000 2024-03-25T19:40:07.431 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-03-25 18:30:15.528769546 +0000 2024-03-25T19:40:07.431 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-03-25 18:30:15.528769546 +0000 2024-03-25T19:40:07.431 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-03-25 18:30:15.528769546 +0000 2024-03-25T19:40:07.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:40:07.494 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-03-25T19:40:07.495 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-03-25T19:40:07.495 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.4929e-05 s, 5.4 MB/s 2024-03-25T19:40:07.496 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:40:07.551 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:07.551 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:07.551 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:40:07.568 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:40:07.568 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 2024-03-25T19:40:07.625 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:40:07.625 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:07.625 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:40:07.626 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:07.626 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:07.626 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-25 18:36:12.280887364 +0000 2024-03-25T19:40:07.626 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-25 18:30:14.542886890 +0000 2024-03-25T19:40:07.626 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-25 18:30:14.542886890 +0000 2024-03-25T19:40:07.626 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-25 18:30:14.542886890 +0000 2024-03-25T19:40:07.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:40:07.694 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-25T19:40:07.694 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-25T19:40:07.694 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000198645 s, 2.6 MB/s 2024-03-25T19:40:07.695 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:40:07.752 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-25 18:36:12.280887364 +0000 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-25 18:30:14.889876019 +0000 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-25 18:30:14.889876019 +0000 2024-03-25T19:40:07.808 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-25 18:30:14.889876019 +0000 2024-03-25T19:40:07.809 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:40:07.875 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-25T19:40:07.875 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-25T19:40:07.875 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000191373 s, 2.7 MB/s 2024-03-25T19:40:07.876 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:40:07.933 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 2024-03-25T19:40:07.988 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:40:07.988 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:07.988 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:40:07.988 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:07.988 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:07.988 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-25 18:36:12.281887333 +0000 2024-03-25T19:40:07.988 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-25 18:30:15.223865554 +0000 2024-03-25T19:40:07.989 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-25 18:30:15.223865554 +0000 2024-03-25T19:40:07.989 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-25 18:30:15.223865554 +0000 2024-03-25T19:40:07.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:40:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-25T19:40:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-25T19:40:08.055 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000194091 s, 2.6 MB/s 2024-03-25T19:40:08.056 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:40:08.113 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2024-03-25 18:36:12.281887333 +0000 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2024-03-25 18:30:15.549855340 +0000 2024-03-25T19:40:08.168 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2024-03-25 18:30:15.549855340 +0000 2024-03-25T19:40:08.169 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: 2024-03-25 18:30:15.549855340 +0000 2024-03-25T19:40:08.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:40:08.234 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2024-03-25T19:40:08.234 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2024-03-25T19:40:08.234 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000165819 s, 3.1 MB/s 2024-03-25T19:40:08.235 DEBUG:teuthology.orchestra.run.smithi076:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:40:08.290 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:08.290 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:40:08.290 INFO:tasks.ceph:Generating config... 2024-03-25T19:40:08.291 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:40:08.291 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:40:08.291 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:40:08.292 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:40:08.292 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:40:08.292 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:40:08.292 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:40:08.292 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:40:08.292 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:40:08.292 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:40:08.292 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:40:08.292 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:40:08.292 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:40:08.292 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:40:08.292 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:40:08.292 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:40:08.292 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:40:08.293 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:40:08.293 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:40:08.293 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:40:08.293 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:40:08.293 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:40:08.293 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:40:08.293 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:40:08.294 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:40:08.295 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:40:08.295 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:40:08.295 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:40:08.295 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:40:08.295 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:40:08.295 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:40:08.295 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:40:08.295 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:40:08.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:08.353 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:40:08.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:40:08.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:08.469 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.50:6789]'), ('mon.c', '[v1:172.21.15.50:6790]'), ('mon.b', '[v1:172.21.15.76:6789]')] 2024-03-25T19:40:08.470 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.50:6789],[v1:172.21.15.50:6790],[v1:172.21.15.76:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:40:08.471 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:08.471 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:40:08.524 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.50:6789]' --addv c '[v1:172.21.15.50:6790]' --addv b '[v1:172.21.15.76:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:08.608 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = quincy 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 17 (quincy) 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 3 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:08.609 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:40:08.611 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:40:08.625 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2e1d843b-b6f3-4677-9562-2b2edda74f50... 2024-03-25T19:40:08.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:40:08.667 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:40:08.715 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-03-25T19:40:08.715 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2024-03-25T19:40:08.715 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:40:08.715 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2024-03-25T19:40:08.715 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.715 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2024-03-25T19:40:08.715 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:40:08.716 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2024-03-25T19:40:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:40:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:40:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2024-03-25T19:40:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.717 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2024-03-25T19:40:08.718 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.719 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:40:08.719 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.719 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2024-03-25T19:40:08.719 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.719 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2024-03-25T19:40:08.719 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.719 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "[v1:172.21.15.50:6789],[v1:172.21.15.50:6790],[v1:172.21.15.76:6789]" 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: mon election default strategy = 3 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: ms bind msgr2 = False 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 5000 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: ms type = async 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: osd heartbeat grace = 80 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.720 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2024-03-25T19:40:08.721 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2024-03-25T19:40:08.721 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.721 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2024-03-25T19:40:08.721 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2024-03-25T19:40:08.721 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:40:08.721 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.721 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug trim objects = true 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.722 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: bdev async discard = True 2024-03-25T19:40:08.723 INFO:teuthology.orchestra.run.smithi050.stdout: bdev enable discard = True 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore allocator = stupid 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore zero block detection = True 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 1 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 1 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:08.724 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd class default list = * 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd class load list = * 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd client watch timeout = 120 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify cached snaps = True 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify missing on start = True 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd fast shutdown = False 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd max markdown count = 1000 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock override recovery settings = True 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2024-03-25T19:40:08.725 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue = debug_random 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue cut off = debug_random 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: osd shutdown pgref assert = True 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2024-03-25T19:40:08.726 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2024-03-25T19:40:08.727 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:40:08.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon min osdmap epochs = 50 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd crush smoke test = False 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune interval = 2 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune min = 15 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub interval = 300 2024-03-25T19:40:08.729 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on pool no app = False 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: paxos service trim min = 10 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:40:08.730 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2024-03-25T19:40:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-03-25T19:40:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: debug objecter = 20 2024-03-25T19:40:08.731 INFO:teuthology.orchestra.run.smithi050.stdout: debug rados = 20 2024-03-25T19:40:08.731 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2024-03-25T19:40:08.731 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2024-03-25T19:40:08.731 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2024-03-25T19:40:08.733 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2024-03-25T19:40:08.733 INFO:teuthology.orchestra.run.smithi076.stdout: chdir = "" 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: auth supported = cephx 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: filestore xattr use omap = true 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: mon clock drift allowed = 1.000 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: osd crush chooseleaf type = 0 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: auth debug = true 2024-03-25T19:40:08.734 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on old message = true 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: ms die on bug = true 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: mon pg warn max object skew = 0 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default size = 2 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.735 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow primary affinity = true 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd allow pg remap = true 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on no sortbitwise = false 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on too few osds = false 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: mon_allow_pool_size_one = true 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.736 INFO:teuthology.orchestra.run.smithi076.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: osd default data pool replay window = 5 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: mon allow pool delete = true 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: mon cluster log file level = debug 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: debug asserts on shutdown = true 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: mon health detail to clog = false 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: mon host = "[v1:172.21.15.50:6789],[v1:172.21.15.50:6790],[v1:172.21.15.76:6789]" 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: mon client directed command retry = 5 2024-03-25T19:40:08.737 INFO:teuthology.orchestra.run.smithi076.stdout: mon election default strategy = 3 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: ms bind msgr2 = False 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: ms inject socket failures = 5000 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: ms type = async 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: osd heartbeat grace = 80 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout:[osd] 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: osd journal size = 100 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub load threshold = 5.0 2024-03-25T19:40:08.738 INFO:teuthology.orchestra.run.smithi076.stdout: osd scrub max interval = 600 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd recover clone overlap = true 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd recovery max chunk = 1048576 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug shutdown = true 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug op order = true 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify stray on activate = true 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug trim objects = true 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.739 INFO:teuthology.orchestra.run.smithi076.stdout: osd open classes on start = true 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug pg log writeout = true 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: osd map max advance = 10 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: journal zero on create = true 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: filestore apply finisher threads = 3 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: bdev debug aio = true 2024-03-25T19:40:08.740 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug misdirected ops = true 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: bdev async discard = True 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: bdev enable discard = True 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore allocator = stupid 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore block size = 96636764160 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore fsck on mount = True 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: bluestore zero block detection = True 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluefs = 1 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: debug bluestore = 1 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: debug monc = 20 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: debug osd = 20 2024-03-25T19:40:08.741 INFO:teuthology.orchestra.run.smithi076.stdout: debug rocksdb = 10 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd full ratio = 0.9 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd class default list = * 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd class load list = * 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd client watch timeout = 120 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify cached snaps = True 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd debug verify missing on start = True 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:40:08.742 INFO:teuthology.orchestra.run.smithi076.stdout: osd fast shutdown = False 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: osd max markdown count = 1000 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: osd mclock override recovery settings = True 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: osd objectstore = bluestore 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: osd op queue = debug_random 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: osd op queue cut off = debug_random 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: osd shutdown pgref assert = True 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr] 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: debug mgr = 20 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-03-25T19:40:08.743 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: mgr/telemetry/nag = false 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout:[mon] 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: debug mon = 20 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: debug paxos = 20 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: debug auth = 20 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: mon data avail warn = 5 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: mon mgr mkfs grace = 240 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:08.744 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd reporter subtree level = osd 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd prime pg temp = true 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: # 1m isn't quite enough 2024-03-25T19:40:08.745 INFO:teuthology.orchestra.run.smithi076.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon min osdmap epochs = 50 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon osd crush smoke test = False 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon osdmap full prune interval = 2 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon osdmap full prune min = 15 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon scrub interval = 300 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: mon warn on pool no app = False 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: paxos service trim min = 10 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout: 2024-03-25T19:40:08.746 INFO:teuthology.orchestra.run.smithi076.stdout:[client] 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: rgw cache enabled = true 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable ops log = true 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: rgw enable usage log = true 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: debug monc = 20 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: debug ms = 1 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: debug objecter = 20 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout: debug rados = 20 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.a] 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.c] 2024-03-25T19:40:08.747 INFO:teuthology.orchestra.run.smithi076.stdout:[mon.b] 2024-03-25T19:40:08.748 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:40:08.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:40:08.803 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:40:08.803 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:08.803 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:40:08.817 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:08.817 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:40:08.871 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:40:08.871 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:08.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:40:08.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:08.950 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:08.950 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:09.004 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:40:09.004 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:09.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:40:09.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:09.044 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:09.044 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:09.098 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:40:09.099 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:40:09.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:40:09.152 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:40:09.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:40:09.223 INFO:teuthology.orchestra.run.smithi076.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:40:09.226 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:40:09.226 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:40:09.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:40:09.268 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:40:09.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:40:09.325 INFO:teuthology.orchestra.run.smithi076.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:40:09.340 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:40:09.340 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:40:09.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:09.366 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:09.366 INFO:tasks.ceph:role: osd.0 2024-03-25T19:40:09.366 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:40:09.367 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:09.432 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:09.465 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-03-25T19:40:09.467 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-03-25T19:40:09.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:09.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:09.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:09.675 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:09.675 INFO:tasks.ceph:role: osd.1 2024-03-25T19:40:09.675 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:40:09.675 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:09.741 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:09.773 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-03-25T19:40:09.775 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-03-25T19:40:09.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:09.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:09.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:09.989 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:09.989 INFO:tasks.ceph:role: osd.2 2024-03-25T19:40:09.990 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:40:09.990 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:10.055 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:10.055 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:10.055 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:10.055 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:10.055 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:10.055 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:10.055 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:10.056 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:10.056 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:10.056 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:10.087 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-03-25T19:40:10.089 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-03-25T19:40:10.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:10.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:10.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:10.296 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:10.296 INFO:tasks.ceph:role: osd.3 2024-03-25T19:40:10.296 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:40:10.296 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:10.362 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:10.363 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:10.363 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:10.395 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-03-25T19:40:10.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-03-25T19:40:10.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:10.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:10.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:10.635+0000 7f6b6dc0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:40:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:10.635+0000 7f6b6dc0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:40:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:10.635+0000 7f6b6dc0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:10.635+0000 7f6b6dc0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:10.635+0000 7f6b6dc0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:10.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:10.635+0000 7f6b6dc0f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:40:13.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:13.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:13.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:13.594+0000 7f667f00e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:40:13.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:13.594+0000 7f667f00e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:40:13.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:13.594+0000 7f667f00e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:13.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:13.594+0000 7f667f00e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:13.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:13.594+0000 7f667f00e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:13.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:13.595+0000 7f667f00e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:40:16.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:16.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:16.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:16.560+0000 7fdc1b3df640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:40:16.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:16.560+0000 7fdc1b3df640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:40:16.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:16.560+0000 7fdc1b3df640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:16.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:16.560+0000 7fdc1b3df640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:16.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:16.560+0000 7fdc1b3df640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:16.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:16.560+0000 7fdc1b3df640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:40:19.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:19.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:19.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:19.525+0000 7fdd8e32a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:40:19.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:19.525+0000 7fdd8e32a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:40:19.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:19.525+0000 7fdd8e32a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:19.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:19.525+0000 7fdd8e32a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:19.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:19.525+0000 7fdd8e32a640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:19.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:19.525+0000 7fdd8e32a640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:40:22.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:22.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:22.423 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:22.423 INFO:tasks.ceph:role: osd.4 2024-03-25T19:40:22.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:40:22.424 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:22.502 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:22.502 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:22.503 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:22.985 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-03-25T19:40:22.987 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-03-25T19:40:22.987 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:23.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:23.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:23.232 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:23.232 INFO:tasks.ceph:role: osd.5 2024-03-25T19:40:23.232 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:40:23.232 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:23.311 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:23.772 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-03-25T19:40:23.774 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-03-25T19:40:23.774 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:23.874 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:23.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:23.983 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:23.983 INFO:tasks.ceph:role: osd.6 2024-03-25T19:40:23.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:40:23.984 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:24.062 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:24.062 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:24.062 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:24.062 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:24.062 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:24.063 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:24.547 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-03-25T19:40:24.549 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-03-25T19:40:24.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:24.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:24.713 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:24.784 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:24.784 INFO:tasks.ceph:role: osd.7 2024-03-25T19:40:24.784 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:40:24.785 DEBUG:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:40:24.863 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:24.864 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:25.367 INFO:teuthology.orchestra.run.smithi076.stdout:Discarding blocks...Done. 2024-03-25T19:40:25.369 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2024-03-25T19:40:25.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:25.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:25.537 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:25.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:25.644+0000 7f4591224640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:40:25.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:25.645+0000 7f4591224640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:40:25.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:25.645+0000 7f4591224640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:25.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:25.645+0000 7f4591224640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:25.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:25.645+0000 7f4591224640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:40:25.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:25.645+0000 7f4591224640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:40:28.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:28.516 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:28.607+0000 7f46dcc10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:40:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:28.607+0000 7f46dcc10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:40:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:28.607+0000 7f46dcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:28.608+0000 7f46dcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:28.608+0000 7f46dcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:40:28.607 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:28.608+0000 7f46dcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:40:31.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:31.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:31.569 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:31.570+0000 7f92d0149640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:40:31.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:31.570+0000 7f92d0149640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:40:31.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:31.570+0000 7f92d0149640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:31.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:31.570+0000 7f92d0149640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:31.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:31.570+0000 7f92d0149640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:40:31.570 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:31.571+0000 7f92d0149640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:40:34.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:34.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:34.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:34.532+0000 7fc6dae13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:40:34.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:34.533+0000 7fc6dae13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:40:34.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:34.533+0000 7fc6dae13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:34.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:34.533+0000 7fc6dae13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:34.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:34.533+0000 7fc6dae13640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:40:34.532 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:34.533+0000 7fc6dae13640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:40:37.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:40:37.404 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:40:37.404 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:37.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:40:37.436 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:37.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:40:37.507 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:37.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:40:37.577 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:37.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:40:37.648 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:37.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:40:37.718 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:37.718 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:40:37.748 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:37.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:40:37.818 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:37.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:40:37.886 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:37.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:40:37.955 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:37.955 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:40:38.022 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:38.022 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:40:38.039 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:38.040 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:40:38.077 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:40:38.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:38.082 DEBUG:teuthology.orchestra.run.smithi076:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.y] 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAZ0wFmEu8zCRAA3pvdwyF233FYlfE3OMb3aw== 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAa0wFmHmz6JRAAkVavQZkN7g20PJS3SApe3g== 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAd0wFmcfKNIxAAz3SpzfQTVvK1wbmgmyeIVg== 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAg0wFmUOByIRAAwrPrCtNP5iWivEev5ilVAg== 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAj0wFmCetbHxAAPdNrXOw0DQ9GvHZCBlkOcA== 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAZ0wFm1q5sDRAAqVLOQNO8SpvV0swPURYklw== 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2024-03-25T19:40:38.120 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAp0wFmLxl2JhAAx9qAbK41uV66iOBnw8sFTg== 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAs0wFmON08JBAAXqr8IotATNftrr6VTFIfpQ== 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.6] 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAv0wFm1zoHIhAAQ4n+yoAoLWL5v0vNUMCwVA== 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.7] 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAy0wFmAy3JHxAA4kdhPftybLDGniGeE2s87w== 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAZ0wFmipwREBAAnoGFXmAwWoa7VHQZsAtAoQ== 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout:[client.1] 2024-03-25T19:40:38.121 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQAZ0wFmPzN3ExAApgLn2m5gnL4fAgCDPWlxpQ== 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.y] 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAZ0wFmEu8zCRAA3pvdwyF233FYlfE3OMb3aw== 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.0] 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAa0wFmHmz6JRAAkVavQZkN7g20PJS3SApe3g== 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.1] 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAd0wFmcfKNIxAAz3SpzfQTVvK1wbmgmyeIVg== 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.2] 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAg0wFmUOByIRAAwrPrCtNP5iWivEev5ilVAg== 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.3] 2024-03-25T19:40:38.142 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAj0wFmCetbHxAAPdNrXOw0DQ9GvHZCBlkOcA== 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.x] 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAZ0wFm1q5sDRAAqVLOQNO8SpvV0swPURYklw== 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.4] 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAp0wFmLxl2JhAAx9qAbK41uV66iOBnw8sFTg== 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.5] 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAs0wFmON08JBAAXqr8IotATNftrr6VTFIfpQ== 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.6] 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAv0wFm1zoHIhAAQ4n+yoAoLWL5v0vNUMCwVA== 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.7] 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAy0wFmAy3JHxAA4kdhPftybLDGniGeE2s87w== 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAZ0wFmipwREBAAnoGFXmAwWoa7VHQZsAtAoQ== 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout:[client.1] 2024-03-25T19:40:38.143 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAZ0wFmPzN3ExAApgLn2m5gnL4fAgCDPWlxpQ== 2024-03-25T19:40:38.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:38.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:38.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:38.454 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:40:38.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:40:38.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:38.723 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:38.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:38.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:40:38.834 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:40:38.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:38.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:38.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:40:38.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:39.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:39.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:40:39.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:39.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:39.297 DEBUG:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:40:39.324 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:39.326 DEBUG:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:39.379 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:40:39.379 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:39.380 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:40:39.380 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:40:39.382 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:40:39.382 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:39.382 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:40:39.382 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:40:39.384 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:40:39.384 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:40:39.384 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:40:39.384 DEBUG:teuthology.orchestra.run.smithi076:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:40:39.421 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:40:39.421 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:40:39.421 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:40:39.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:40:39.424 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:40:39.424 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:40:39.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:40:39.426 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:40:39.426 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:40:39.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:40:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.529+0000 7f8eaaa4e640 10 monclient: get_monmap_and_config 2024-03-25T19:40:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.529+0000 7f8eaaa4e640 10 monclient: build_initial_monmap 2024-03-25T19:40:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.529+0000 7f8eaaa4e640 10 monclient: monmap: 2024-03-25T19:40:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2024-03-25T19:40:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:39.528 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:39.529905+0000 2024-03-25T19:40:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:39.529905+0000 2024-03-25T19:40:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:40:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:40:39.529 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:40:39.529 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:40:39.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.529+0000 7f8eaaa4e640 1 Processor -- start 2024-03-25T19:40:39.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.529+0000 7f8eaaa4e640 1 -- start start 2024-03-25T19:40:39.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.529+0000 7f8eaaa4e640 10 monclient: init 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea40a9710 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea413c570 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40ab420 con 0x7f8ea413c570 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40b0190 con 0x7f8ea40aafa0 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40abdb0 con 0x7f8ea40a9710 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8eaaa4e640 10 monclient(hunting): authenticate will time out at 4814.794283s 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8ea414a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8ea40ad3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:39.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.530+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8ea40724a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:39.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.731+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8ea40ad3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:39.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.731+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8ea414a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:39.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:39.731+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8ea40724a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:40.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:40.132+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8ea40ad3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:40.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:40.132+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8ea414a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:40.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:40.132+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8ea40724a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:40.933+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8ea40ad3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:40.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:40.933+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8ea414a9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:40.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:40.933+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8ea40724a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient(hunting): tick 2024-03-25T19:40:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:42.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8ea40ad3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8ea40724a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8ea414a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea413c570 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea40a9710 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40abf40 con 0x7f8ea413c570 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e900056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40ab420 con 0x7f8ea40aafa0 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40b0190 con 0x7f8ea40a9710 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea1ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:42.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.530+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.731+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.731+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e900056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:42.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:42.731+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:43.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:43.132+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:43.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:43.132+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e900056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:43.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:43.132+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:43.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:43.933+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e900056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:43.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:43.933+0000 7f8ea8feb640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:43.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:43.933+0000 7f8ea37fe640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:45.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient(hunting): tick 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e900056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea40a9710 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea413c570 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40abdb0 con 0x7f8ea413c570 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40abf40 con 0x7f8ea40aafa0 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40ab420 con 0x7f8ea40a9710 2024-03-25T19:40:45.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea1ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.530+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.531+0000 7f8ea8feb640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:45.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.531+0000 7f8ea37fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:45.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.732+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:45.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.732+0000 7f8ea37fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:45.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:45.732+0000 7f8ea8feb640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:46.133+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:46.133+0000 7f8ea37fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:46.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:46.133+0000 7f8ea8feb640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:46.401 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:00:06.871 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:46.710 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:00:07.181 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:46.892 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:07.398 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:46.892 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.398 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:46.934+0000 7f8ea3fff640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6790/0 2024-03-25T19:40:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:46.934+0000 7f8ea37fe640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.50:6789/0 2024-03-25T19:40:46.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:46.934+0000 7f8ea8feb640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.76:6789/0 2024-03-25T19:40:47.219 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:00:07.725 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:47.220 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:00:07.726 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:40:48.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient(hunting): tick 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 1 -- >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea413c570 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea40a9710 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40b0190 con 0x7f8ea413c570 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40abdb0 con 0x7f8ea40aafa0 2024-03-25T19:40:48.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40abf40 con 0x7f8ea40a9710 2024-03-25T19:40:48.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.531+0000 7f8ea1ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:48.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.586+0000 7f8ea8feb640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 0x7f8e90010570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:44474/0 (socket says 172.21.15.50:44474) 2024-03-25T19:40:48.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.586+0000 7f8ea8feb640 1 -- 172.21.15.50:0/2271320821 learned_addr learned my addr 172.21.15.50:0/2271320821 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:40:48.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.881+0000 7f8ea3fff640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T19:40:48.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.881+0000 7f8ea3fff640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:40:48.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.881+0000 7f8ea3fff640 1 --1- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 0x7f8e9000fe60 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:40:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.917+0000 7f8ea37fe640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T19:40:48.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.917+0000 7f8ea37fe640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:40:48.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:48.917+0000 7f8ea37fe640 1 --1- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 0x7f8e9000c6a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient(hunting): tick 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6790/0 conn(0x7f8ea40a9710 legacy=0x7f8e9000c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90010570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:51.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f8e90005740 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea413c570 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea40ab420 con 0x7f8ea413c570 2024-03-25T19:40:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e90006c60 con 0x7f8ea40aafa0 2024-03-25T19:40:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9000e760 con 0x7f8e90005740 2024-03-25T19:40:51.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:51.531+0000 7f8ea1ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:54.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient(hunting): tick 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6790/0 conn(0x7f8e90005740 legacy=0x7f8e90008ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90010570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6789/0 conn(0x7f8ea413c570 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9000e8f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f8e90005740 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9000f9a0 con 0x7f8e9000e8f0 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9000f440 con 0x7f8ea40aafa0 2024-03-25T19:40:54.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9000f620 con 0x7f8e90005740 2024-03-25T19:40:54.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:54.531+0000 7f8ea1ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.531+0000 7f8ea1ffb640 10 monclient(hunting): tick 2024-03-25T19:40:57.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 1 monclient(hunting): continuing hunt 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6790/0 conn(0x7f8e90005740 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90010570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6789/0 conn(0x7f8e9000e8f0 legacy=0x7f8e9000ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9000e8f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f8e90005740 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e90008eb0 con 0x7f8e9000e8f0 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9000a050 con 0x7f8ea40aafa0 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e9000a230 con 0x7f8e90005740 2024-03-25T19:40:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.532+0000 7f8ea1ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:57.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.735+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006628347 0 0) 0x7f8e90008eb0 con 0x7f8e9000e8f0 2024-03-25T19:40:57.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.735+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.735+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.735+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.735+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4104 2024-03-25T19:40:57.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.735+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e8c003520 con 0x7f8e9000e8f0 2024-03-25T19:40:57.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.793+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282500550 0 0) 0x7f8e9000a230 con 0x7f8e90005740 2024-03-25T19:40:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.793+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.793+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.793+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.793+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:40:57.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.793+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e90008eb0 con 0x7f8e90005740 2024-03-25T19:40:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.810+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204561766 0 0) 0x7f8e9000a050 con 0x7f8ea40aafa0 2024-03-25T19:40:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.810+0000 7f8ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.810+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.810+0000 7f8ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.810+0000 7f8ea2ffd640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:40:57.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.810+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9000a230 con 0x7f8ea40aafa0 2024-03-25T19:40:57.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315898358 0 0) 0x7f8e8c003520 con 0x7f8e9000e8f0 2024-03-25T19:40:57.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6790/0 conn(0x7f8e90005740 legacy=0x7f8e9000fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.76:6789/0 conn(0x7f8ea40aafa0 legacy=0x7f8e90010570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:57.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 1 monclient: found mon.noname-a 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e90012f50 con 0x7f8e9000e8f0 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8e98004080 con 0x7f8e9000e8f0 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8eaaa4e640 5 monclient: authenticate success, global_id 4104 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8eaaa4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.848+0000 7f8ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.849+0000 7f8ea2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.849+0000 7f8ea2ffd640 10 monclient: dump: 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:40:57.849 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:40:57.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.849+0000 7f8eaaa4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:57.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.859+0000 7f8ea2ffd640 1 -- 172.21.15.50:0/2271320821 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e98004250 con 0x7f8e9000e8f0 2024-03-25T19:40:57.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.859+0000 7f8ea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:57.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.859+0000 7f8eaaa4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:57.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.859+0000 7f8eaaa4e640 4 set_mon_vals no callback set 2024-03-25T19:40:57.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.859+0000 7f8eaaa4e640 10 monclient: shutdown 2024-03-25T19:40:57.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.859+0000 7f8eaaa4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:57.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.859+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2271320821 >> v1:172.21.15.50:6789/0 conn(0x7f8e9000e8f0 legacy=0x7f8e9000ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.860+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2271320821 shutdown_connections 2024-03-25T19:40:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.860+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2271320821 >> 172.21.15.50:0/2271320821 conn(0x7f8ea406b560 msgr2=0x7f8ea40b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.860+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2271320821 shutdown_connections 2024-03-25T19:40:57.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.860+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2271320821 wait complete. 2024-03-25T19:40:57.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.860+0000 7f8eaaa4e640 10 monclient: build_initial_monmap 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.860+0000 7f8eaaa4e640 10 monclient: monmap: 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.861+0000 7f8eaaa4e640 1 librados: starting msgr at 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.861+0000 7f8eaaa4e640 1 librados: starting objecter 2024-03-25T19:40:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.861+0000 7f8eaaa4e640 1 Processor -- start 2024-03-25T19:40:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.861+0000 7f8eaaa4e640 1 -- start start 2024-03-25T19:40:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.861+0000 7f8eaaa4e640 1 librados: setting wanted keys 2024-03-25T19:40:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.861+0000 7f8eaaa4e640 1 librados: calling monclient init 2024-03-25T19:40:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.861+0000 7f8eaaa4e640 10 monclient: init 2024-03-25T19:40:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:57.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 10 monclient(hunting): picked mon.noname-b con 0x7f8e90005740 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 10 monclient(hunting): picked mon.noname-a con 0x7f8e9000e8f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ea40aafa0 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea41432c0 con 0x7f8e9000e8f0 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea41d8f80 con 0x7f8e90005740 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea41d8af0 con 0x7f8ea40aafa0 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8eaaa4e640 10 monclient(hunting): authenticate will time out at 4833.125809s 2024-03-25T19:40:57.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8ea8feb640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f8ea40aafa0 0x7f8ea40707f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:40820/0 (socket says 172.21.15.50:40820) 2024-03-25T19:40:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8ea8feb640 1 -- 172.21.15.50:0/2339472017 learned_addr learned my addr 172.21.15.50:0/2339472017 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:40:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8ea0ff9640 10 client.?.objecter ms_handle_connect 0x7f8ea40aafa0 2024-03-25T19:40:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8ea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8ea0ff9640 10 client.?.objecter ms_handle_connect 0x7f8e90005740 2024-03-25T19:40:57.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.862+0000 7f8ea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.900+0000 7f8ea0ff9640 10 client.?.objecter ms_handle_connect 0x7f8e9000e8f0 2024-03-25T19:40:57.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.900+0000 7f8ea0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352348336 0 0) 0x7f8ea41432c0 con 0x7f8e9000e8f0 2024-03-25T19:40:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): my global_id is 4113 2024-03-25T19:40:57.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e9000a460 con 0x7f8e9000e8f0 2024-03-25T19:40:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815551127 0 0) 0x7f8ea41d8af0 con 0x7f8ea40aafa0 2024-03-25T19:40:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:40:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.926+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea41432c0 con 0x7f8ea40aafa0 2024-03-25T19:40:57.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.928+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833725565 0 0) 0x7f8ea41d8f80 con 0x7f8e90005740 2024-03-25T19:40:57.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.928+0000 7f8ea0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:57.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.928+0000 7f8ea0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:57.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.928+0000 7f8ea0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:57.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.928+0000 7f8ea0ff9640 10 monclient(hunting): my global_id is 4111 2024-03-25T19:40:57.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.928+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea41d8af0 con 0x7f8e90005740 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.953+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675112416 0 0) 0x7f8e9000a460 con 0x7f8e9000e8f0 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.953+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ea41d8f80 con 0x7f8e9000e8f0 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.953+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8e980045a0 con 0x7f8e9000e8f0 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.953+0000 7f8ea0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.953+0000 7f8ea0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.953+0000 7f8ea0ff9640 10 monclient(hunting): dump: 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:57.953 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:40:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:57.954 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:40:57.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.953+0000 7f8ea0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1628457488 0 0) 0x7f8ea41d8f80 con 0x7f8e9000e8f0 2024-03-25T19:40:57.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 >> v1:172.21.15.50:6790/0 conn(0x7f8ea40aafa0 legacy=0x7f8ea40707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 >> v1:172.21.15.76:6789/0 conn(0x7f8e90005740 legacy=0x7f8ea41d94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 1 monclient: found mon.a 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea41d8c80 con 0x7f8e9000e8f0 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8ea0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8eaaa4e640 5 monclient: authenticate success, global_id 4113 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8eaaa4e640 10 monclient: _renew_subs 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8eaaa4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea4071470 con 0x7f8e9000e8f0 2024-03-25T19:40:57.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.962+0000 7f8eaaa4e640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.963+0000 7f8eaaa4e640 10 monclient: _renew_subs 2024-03-25T19:40:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.963+0000 7f8eaaa4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.963+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea4070f00 con 0x7f8e9000e8f0 2024-03-25T19:40:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.963+0000 7f8eaaa4e640 1 librados: init done 2024-03-25T19:40:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.964+0000 7f8eaaa4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.964+0000 7f8eaaa4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:57.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.964+0000 7f8eaaa4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:57.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:57.964+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea4143070 con 0x7f8e9000e8f0 2024-03-25T19:40:58.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e980049b0 con 0x7f8e9000e8f0 2024-03-25T19:40:58.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:58.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8e98004dc0 con 0x7f8e9000e8f0 2024-03-25T19:40:58.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 10 monclient: dump: 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8e98005370 con 0x7f8e9000e8f0 2024-03-25T19:40:58.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2083340766 0 0) 0x7f8e98006840 con 0x7f8e9000e8f0 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea27fc640 4 set_mon_vals no callback set 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 10 client.4113.objecter ms_dispatch 0x7f8ea4000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8e98006af0 con 0x7f8e9000e8f0 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.225+0000 7f8ea0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:58.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.226+0000 7f8ea0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.308+0000 7f8eaaa4e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.308+0000 7f8eaaa4e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.308+0000 7f8eaaa4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:58.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.308+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f8ea4143070 con 0x7f8e9000e8f0 2024-03-25T19:40:58.461 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:58.462+0000 7f4acb62c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.462+0000 7f8ea0ff9640 1 -- 172.21.15.50:0/2339472017 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f8e98005930 con 0x7f8e9000e8f0 2024-03-25T19:40:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.462+0000 7f8ea0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:40:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.462+0000 7f8ea0ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:40:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2024-03-25T19:40:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 10 librados: watch_flush enter 2024-03-25T19:40:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 10 librados: watch_flush exit 2024-03-25T19:40:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T19:40:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 10 client.4113.objecter successfully canceled tick 2024-03-25T19:40:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 10 monclient: shutdown 2024-03-25T19:40:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:58.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 >> v1:172.21.15.50:6789/0 conn(0x7f8e9000e8f0 legacy=0x7f8ea41d55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 shutdown_connections 2024-03-25T19:40:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 >> 172.21.15.50:0/2339472017 conn(0x7f8ea406b560 msgr2=0x7f8ea413a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 shutdown_connections 2024-03-25T19:40:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 1 -- 172.21.15.50:0/2339472017 wait complete. 2024-03-25T19:40:58.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:40:58.463+0000 7f8eaaa4e640 1 librados: shutdown 2024-03-25T19:40:58.475 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:40:58.475 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:40:58.475 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:58.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:40:58.503 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:58.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:40:58.524 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:58.524+0000 7f4acb62c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:58.526 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:58.527+0000 7fe64582e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:40:58.550 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:58.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:40:58.580 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:58.581+0000 7fe64582e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:40:58.616 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-03-25T19:40:58.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:40:58.683 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:58.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:40:58.713 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:58.713 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:40:58.779 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:58.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:40:58.847 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2024-03-25T19:40:58.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:40:58.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new 227bd158-cf26-4f25-8ba8-aa8da6d6ba2d 0 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.054+0000 7f53fbe0f640 10 monclient: get_monmap_and_config 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.054+0000 7f53fbe0f640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.054+0000 7f53fbe0f640 10 monclient: monmap: 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:59.055168+0000 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:59.055168+0000 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.054 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.055 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:59.055 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:40:59.055 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:40:59.055 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.055+0000 7f53fbe0f640 1 Processor -- start 2024-03-25T19:40:59.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.055+0000 7f53fbe0f640 1 -- start start 2024-03-25T19:40:59.055 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.055+0000 7f53fbe0f640 10 monclient: init 2024-03-25T19:40:59.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:59.056 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 10 monclient(hunting): picked mon.noname-a con 0x7f53f40b4700 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 10 monclient(hunting): picked mon.noname-b con 0x7f53f4142a20 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 10 monclient(hunting): picked mon.noname-c con 0x7f53f40a96b0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f40a9d60 con 0x7f53f40b4700 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f413ff10 con 0x7f53f40a96b0 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f40aa1f0 con 0x7f53f4142a20 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53fbe0f640 10 monclient(hunting): authenticate will time out at 4823.201869s 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53f9bab640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f53f40b4700 0x7f53f4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52338/0 (socket says 172.21.15.76:52338) 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53f9bab640 1 -- 172.21.15.76:0/3470591150 learned_addr learned my addr 172.21.15.76:0/3470591150 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:40:59.057 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.056+0000 7f53f93aa640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f53f4142a20 0x7f53f413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.76:32926/0 (socket says 172.21.15.76:32926) 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320012483 0 0) 0x7f53f413ff10 con 0x7f53f40a96b0 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e0003520 con 0x7f53f40a96b0 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581040619 0 0) 0x7f53f40a9d60 con 0x7f53f40b4700 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:40:59.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f413ff10 con 0x7f53f40b4700 2024-03-25T19:40:59.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578218774 0 0) 0x7f53f40aa1f0 con 0x7f53f4142a20 2024-03-25T19:40:59.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:40:59.060 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.059+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f40a9d60 con 0x7f53f4142a20 2024-03-25T19:40:59.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2105083448 0 0) 0x7f53e0003520 con 0x7f53f40a96b0 2024-03-25T19:40:59.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 >> v1:172.21.15.50:6790/0 conn(0x7f53f4142a20 legacy=0x7f53f413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 >> v1:172.21.15.50:6789/0 conn(0x7f53f40b4700 legacy=0x7f53f4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 1 monclient: found mon.noname-c 2024-03-25T19:40:59.062 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f40b4ae0 con 0x7f53f40a96b0 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f53ec001a10 con 0x7f53f40a96b0 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53fbe0f640 5 monclient: authenticate success, global_id 4114 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53fbe0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53f8ba9640 10 monclient: dump: 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.063 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.064 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.064 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.064 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.064 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.064 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.064 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.063+0000 7f53fbe0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:59.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53ec001b70 con 0x7f53f40a96b0 2024-03-25T19:40:59.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53f8ba9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53f8ba9640 1 -- 172.21.15.76:0/3470591150 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f53ec001f80 con 0x7f53f40a96b0 2024-03-25T19:40:59.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53f8ba9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53f8ba9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53f8ba9640 10 monclient: dump: 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53fbe0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:59.089 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.089+0000 7f53fbe0f640 4 set_mon_vals no callback set 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 10 monclient: shutdown 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3470591150 >> v1:172.21.15.76:6789/0 conn(0x7f53f40a96b0 legacy=0x7f53f413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3470591150 shutdown_connections 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3470591150 >> 172.21.15.76:0/3470591150 conn(0x7f53f40668e0 msgr2=0x7f53f4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3470591150 shutdown_connections 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3470591150 wait complete. 2024-03-25T19:40:59.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.090+0000 7f53fbe0f640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 10 monclient: monmap: 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 librados: starting msgr at 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 librados: starting objecter 2024-03-25T19:40:59.093 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 Processor -- start 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 -- start start 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 librados: setting wanted keys 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 librados: calling monclient init 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 10 monclient: init 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 10 monclient(hunting): picked mon.noname-b con 0x7f53f40a96b0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 10 monclient(hunting): picked mon.noname-a con 0x7f53f40b4700 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 10 monclient(hunting): picked mon.noname-c con 0x7f53f4142a20 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f41365f0 con 0x7f53f40b4700 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.091+0000 7f53fbe0f640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f4137fd0 con 0x7f53f40a96b0 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53fbe0f640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f41382b0 con 0x7f53f4142a20 2024-03-25T19:40:59.094 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53fbe0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53fbe0f640 10 monclient(hunting): authenticate will time out at 4823.237676s 2024-03-25T19:40:59.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53f93aa640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f53f40b4700 0x7f53f41389b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52346/0 (socket says 172.21.15.76:52346) 2024-03-25T19:40:59.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53f93aa640 1 -- 172.21.15.76:0/3962002164 learned_addr learned my addr 172.21.15.76:0/3962002164 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:40:59.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53ea7fc640 10 client.?.objecter ms_handle_connect 0x7f53f40b4700 2024-03-25T19:40:59.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53ea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53ea7fc640 10 client.?.objecter ms_handle_connect 0x7f53f40a96b0 2024-03-25T19:40:59.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.092+0000 7f53ea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.094+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715516790 0 0) 0x7f53f4137fd0 con 0x7f53f40a96b0 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.094+0000 7f53ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.094+0000 7f53ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.094+0000 7f53ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.094+0000 7f53ea7fc640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.094+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d8004eb0 con 0x7f53f40a96b0 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.095+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825120065 0 0) 0x7f53f41365f0 con 0x7f53f40b4700 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.095+0000 7f53ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.095+0000 7f53ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.095+0000 7f53ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.095+0000 7f53ea7fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:40:59.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.095+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f4137fd0 con 0x7f53f40b4700 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.102+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952328137 0 0) 0x7f53f4137fd0 con 0x7f53f40b4700 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.102+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53f41365f0 con 0x7f53f40b4700 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.102+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f53f0001970 con 0x7f53f40b4700 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.102+0000 7f53ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.102+0000 7f53ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.102+0000 7f53ea7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.101 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.102+0000 7f53ea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:59.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.105+0000 7f53ea7fc640 10 client.?.objecter ms_handle_connect 0x7f53f4142a20 2024-03-25T19:40:59.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.105+0000 7f53ea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2915875662 0 0) 0x7f53f41365f0 con 0x7f53f40b4700 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 >> v1:172.21.15.50:6790/0 conn(0x7f53f4142a20 legacy=0x7f53f41e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 >> v1:172.21.15.76:6789/0 conn(0x7f53f40a96b0 legacy=0x7f53f4134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 1 monclient: found mon.a 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f4138590 con 0x7f53f40b4700 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:40:59.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 10 monclient: _renew_subs 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f4136820 con 0x7f53f40b4700 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 10 monclient: _renew_subs 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f4135ee0 con 0x7f53f40b4700 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.107+0000 7f53fbe0f640 1 librados: init done 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.108+0000 7f53fbe0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.108+0000 7f53fbe0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.108+0000 7f53fbe0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.108 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.108+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f404f730 con 0x7f53f40b4700 2024-03-25T19:40:59.116 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.117+0000 7f4acb62c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:59.188 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.189+0000 7f4acb62c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:59.202 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.204+0000 7fe64582e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53f00047e0 con 0x7f53f40b4700 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f53f0004bb0 con 0x7f53f40b4700 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ea7fc640 10 monclient: dump: 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.230 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.231 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.231 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.231 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.231 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.231 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f53f0005160 con 0x7f53f40b4700 2024-03-25T19:40:59.231 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.230+0000 7f53ebfff640 4 set_mon_vals no callback set 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.231+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 495519304 0 0) 0x7f53f0006560 con 0x7f53f40b4700 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.231+0000 7f53ea7fc640 10 client.4119.objecter ms_dispatch 0x7f53f4000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.231+0000 7f53ea7fc640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.231+0000 7f53ea7fc640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.231+0000 7f53ea7fc640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.231+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f53f00068b0 con 0x7f53f40b4700 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.232+0000 7f53ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.232 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.232+0000 7f53ea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:59.235 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.235+0000 7f4acb62c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:59.258 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.259+0000 7fe64582e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:40:59.290 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.291+0000 7f4acb62c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.304+0000 7f53fbe0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "227bd158-cf26-4f25-8ba8-aa8da6d6ba2d", "id": 0}] 2024-03-25T19:40:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.305+0000 7f53fbe0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "227bd158-cf26-4f25-8ba8-aa8da6d6ba2d", "id": 0}] 2024-03-25T19:40:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.305+0000 7f53fbe0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.304 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.305+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd new", "uuid": "227bd158-cf26-4f25-8ba8-aa8da6d6ba2d", "id": 0} v 0) v1 -- 0x7f53f40b74e0 con 0x7f53f40b4700 2024-03-25T19:40:59.304 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.305+0000 7fe64582e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:40:59.335 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.336+0000 7f4acb62c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:59.359 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.360+0000 7fe64582e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:40:59.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.395+0000 7f53ea7fc640 1 -- 172.21.15.76:0/3962002164 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "227bd158-cf26-4f25-8ba8-aa8da6d6ba2d", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3367602994 0 732707515) 0x7f53f0006a50 con 0x7f53f40b4700 2024-03-25T19:40:59.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.395+0000 7f53ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "227bd158-cf26-4f25-8ba8-aa8da6d6ba2d", "id": 0}] 2024-03-25T19:40:59.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.395+0000 7f53ea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:59.395 INFO:teuthology.orchestra.run.smithi076.stdout:0 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 10 librados: watch_flush enter 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 10 librados: watch_flush exit 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 10 monclient: shutdown 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 >> v1:172.21.15.50:6789/0 conn(0x7f53f40b4700 legacy=0x7f53f41389b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 shutdown_connections 2024-03-25T19:40:59.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 >> 172.21.15.76:0/3962002164 conn(0x7f53f40668e0 msgr2=0x7f53f40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:59.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 shutdown_connections 2024-03-25T19:40:59.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 1 -- 172.21.15.76:0/3962002164 wait complete. 2024-03-25T19:40:59.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.396+0000 7f53fbe0f640 1 librados: shutdown 2024-03-25T19:40:59.403 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.404+0000 7fe64582e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:40:59.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new 6e59a3aa-99f7-4d22-b629-20d3efbb0038 1 2024-03-25T19:40:59.459 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.459+0000 7f4acb62c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:59.507 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.508+0000 7f8d9fc33640 10 monclient: get_monmap_and_config 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.508+0000 7f8d9fc33640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.508+0000 7f8d9fc33640 10 monclient: monmap: 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:59.508934+0000 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:59.508934+0000 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:40:59.508 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:40:59.509 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.509 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 1 Processor -- start 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 1 -- start start 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient: init 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient(hunting): picked mon.noname-a con 0x7f8d980b4700 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient(hunting): picked mon.noname-b con 0x7f8d98142a20 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient(hunting): picked mon.noname-c con 0x7f8d980a96b0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980a9d60 con 0x7f8d980b4700 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9813ff10 con 0x7f8d980a96b0 2024-03-25T19:40:59.510 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980aa1f0 con 0x7f8d98142a20 2024-03-25T19:40:59.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.509+0000 7f8d9fc33640 10 monclient(hunting): authenticate will time out at 4823.655553s 2024-03-25T19:40:59.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.510+0000 7f8d9d9cf640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8d980b4700 0x7f8d98142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52350/0 (socket says 172.21.15.76:52350) 2024-03-25T19:40:59.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.510+0000 7f8d9d9cf640 1 -- 172.21.15.76:0/992858671 learned_addr learned my addr 172.21.15.76:0/992858671 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:40:59.511 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959005718 0 0) 0x7f8d9813ff10 con 0x7f8d980a96b0 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d84003520 con 0x7f8d980a96b0 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640267748 0 0) 0x7f8d980a9d60 con 0x7f8d980b4700 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:40:59.512 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.512+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d9813ff10 con 0x7f8d980b4700 2024-03-25T19:40:59.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.513+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451716434 0 0) 0x7f8d980aa1f0 con 0x7f8d98142a20 2024-03-25T19:40:59.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.513+0000 7f8d9c9cd640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.513+0000 7f8d9c9cd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.513+0000 7f8d9c9cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.513+0000 7f8d9c9cd640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:40:59.513 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.513+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d980a9d60 con 0x7f8d98142a20 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.515+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3750997616 0 0) 0x7f8d84003520 con 0x7f8d980a96b0 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 >> v1:172.21.15.50:6790/0 conn(0x7f8d98142a20 legacy=0x7f8d9813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 >> v1:172.21.15.50:6789/0 conn(0x7f8d980b4700 legacy=0x7f8d98142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 1 monclient: found mon.noname-c 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:40:59.515 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980b4ae0 con 0x7f8d980a96b0 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8d90001a10 con 0x7f8d980a96b0 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9c9cd640 10 monclient: dump: 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.516 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.517 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.517 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.517 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.517 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9fc33640 5 monclient: authenticate success, global_id 4120 2024-03-25T19:40:59.517 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.516+0000 7f8d9fc33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:59.520 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d90001b70 con 0x7f8d980a96b0 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9c9cd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9c9cd640 1 -- 172.21.15.76:0/992858671 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8d90001f80 con 0x7f8d980a96b0 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9c9cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9c9cd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9c9cd640 10 monclient: dump: 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.521 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9fc33640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9fc33640 4 set_mon_vals no callback set 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9fc33640 10 monclient: shutdown 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9fc33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:59.522 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.518+0000 7f8d9fc33640 1 -- 172.21.15.76:0/992858671 >> v1:172.21.15.76:6789/0 conn(0x7f8d980a96b0 legacy=0x7f8d9813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.519+0000 7f8d9fc33640 1 -- 172.21.15.76:0/992858671 shutdown_connections 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.519+0000 7f8d9fc33640 1 -- 172.21.15.76:0/992858671 >> 172.21.15.76:0/992858671 conn(0x7f8d980668e0 msgr2=0x7f8d98066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.519+0000 7f8d9fc33640 1 -- 172.21.15.76:0/992858671 shutdown_connections 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.519+0000 7f8d9fc33640 1 -- 172.21.15.76:0/992858671 wait complete. 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.519+0000 7f8d9fc33640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.519+0000 7f8d9fc33640 10 monclient: monmap: 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:40:59.523 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 librados: starting msgr at 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 librados: starting objecter 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 Processor -- start 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 -- start start 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 librados: setting wanted keys 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 librados: calling monclient init 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient: init 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient(hunting): picked mon.noname-c con 0x7f8d980a96b0 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient(hunting): picked mon.noname-b con 0x7f8d980b4700 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:59.524 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient(hunting): picked mon.noname-a con 0x7f8d98142a20 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9813b3e0 con 0x7f8d98142a20 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980ae520 con 0x7f8d980b4700 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980ab880 con 0x7f8d980a96b0 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.520+0000 7f8d9fc33640 10 monclient(hunting): authenticate will time out at 4823.666518s 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d9d9cf640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f8d980a96b0 0x7f8d980ac810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.76:32944/0 (socket says 172.21.15.76:32944) 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d9d9cf640 1 -- 172.21.15.76:0/3042153220 learned_addr learned my addr 172.21.15.76:0/3042153220 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d8e7fc640 10 client.?.objecter ms_handle_connect 0x7f8d980b4700 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d8e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d8e7fc640 10 client.?.objecter ms_handle_connect 0x7f8d980a96b0 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d8e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.525 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d8e7fc640 10 client.?.objecter ms_handle_connect 0x7f8d98142a20 2024-03-25T19:40:59.526 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.521+0000 7f8d8e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93636110 0 0) 0x7f8d980ae520 con 0x7f8d980b4700 2024-03-25T19:40:59.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:40:59.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c004eb0 con 0x7f8d980b4700 2024-03-25T19:40:59.528 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806476367 0 0) 0x7f8d9813b3e0 con 0x7f8d98142a20 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.523+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d980ae520 con 0x7f8d98142a20 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.524+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436952921 0 0) 0x7f8d980ab880 con 0x7f8d980a96b0 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.524+0000 7f8d8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.524+0000 7f8d8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.524+0000 7f8d8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.524+0000 7f8d8e7fc640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:40:59.529 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.524+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d9813b3e0 con 0x7f8d980a96b0 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197366821 0 0) 0x7f8d7c004eb0 con 0x7f8d980b4700 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d980ab880 con 0x7f8d980b4700 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8d880018f0 con 0x7f8d980b4700 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.530 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3844373476 0 0) 0x7f8d980ae520 con 0x7f8d98142a20 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d7c004eb0 con 0x7f8d98142a20 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8d90001e80 con 0x7f8d98142a20 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.528+0000 7f8d8e7fc640 10 monclient(hunting): dump: 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.531 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.532 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.532 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.530+0000 7fe64582e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3381364130 0 0) 0x7f8d7c004eb0 con 0x7f8d98142a20 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 >> v1:172.21.15.50:6790/0 conn(0x7f8d980a96b0 legacy=0x7f8d980ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 >> v1:172.21.15.76:6789/0 conn(0x7f8d980b4700 legacy=0x7f8d98139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 1 monclient: found mon.a 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980aba10 con 0x7f8d98142a20 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 10 monclient: _renew_subs 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9813a490 con 0x7f8d98142a20 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 10 monclient: _renew_subs 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d980ad430 con 0x7f8d98142a20 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.532+0000 7f8d9fc33640 1 librados: init done 2024-03-25T19:40:59.535 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.533+0000 7f8d9fc33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.533+0000 7f8d9fc33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.533+0000 7f8d9fc33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.536 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.533+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9804f730 con 0x7f8d98142a20 2024-03-25T19:40:59.536 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.535+0000 7f4acb62c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:59.583 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.583+0000 7f4acb62c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:59.599 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.600+0000 7fe64582e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d90001fe0 con 0x7f8d98142a20 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8d90003dc0 con 0x7f8d98142a20 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8e7fc640 10 monclient: dump: 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.636 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8ffff640 4 set_mon_vals no callback set 2024-03-25T19:40:59.637 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.637+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8d900042b0 con 0x7f8d98142a20 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 4235230930 0 0) 0x7f8d90005670 con 0x7f8d98142a20 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 10 client.4125.objecter ms_dispatch 0x7f8d98000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:40:59.638 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8d900059e0 con 0x7f8d98142a20 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.639 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.638+0000 7f8d8e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:40:59.639 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.639+0000 7f4acb62c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:59.645 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.646+0000 7fe64582e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:40:59.687 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:40:59.688+0000 7f4acb62c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:59.699 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.701+0000 7fe64582e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:40:59.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.712+0000 7f8d9fc33640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6e59a3aa-99f7-4d22-b629-20d3efbb0038", "id": 1}] 2024-03-25T19:40:59.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.712+0000 7f8d9fc33640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6e59a3aa-99f7-4d22-b629-20d3efbb0038", "id": 1}] 2024-03-25T19:40:59.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.712+0000 7f8d9fc33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:40:59.711 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.712+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6e59a3aa-99f7-4d22-b629-20d3efbb0038", "id": 1} v 0) v1 -- 0x7f8d98142220 con 0x7f8d98142a20 2024-03-25T19:40:59.748 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:40:59.749+0000 7fe64582e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:40:59.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.765+0000 7f8d8e7fc640 1 -- 172.21.15.76:0/3042153220 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6e59a3aa-99f7-4d22-b629-20d3efbb0038", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1216455347 0 940483276) 0x7f8d90004720 con 0x7f8d98142a20 2024-03-25T19:40:59.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.765+0000 7f8d8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6e59a3aa-99f7-4d22-b629-20d3efbb0038", "id": 1}] 2024-03-25T19:40:59.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.765+0000 7f8d8e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:40:59.765 INFO:teuthology.orchestra.run.smithi076.stdout:1 2024-03-25T19:40:59.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 10 librados: watch_flush enter 2024-03-25T19:40:59.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 10 librados: watch_flush exit 2024-03-25T19:40:59.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:40:59.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:40:59.766 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 10 monclient: shutdown 2024-03-25T19:40:59.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:59.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 >> v1:172.21.15.50:6789/0 conn(0x7f8d98142a20 legacy=0x7f8d980b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 shutdown_connections 2024-03-25T19:40:59.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 >> 172.21.15.76:0/3042153220 conn(0x7f8d980668e0 msgr2=0x7f8d9813c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:59.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.766+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 shutdown_connections 2024-03-25T19:40:59.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.767+0000 7f8d9fc33640 1 -- 172.21.15.76:0/3042153220 wait complete. 2024-03-25T19:40:59.767 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.767+0000 7f8d9fc33640 1 librados: shutdown 2024-03-25T19:40:59.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new ff7c8b95-7815-449d-a015-a11978f86ec7 2 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.876+0000 7ffb2ebfb640 10 monclient: get_monmap_and_config 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.876+0000 7ffb2ebfb640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.876+0000 7ffb2ebfb640 10 monclient: monmap: 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:59.877266+0000 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:59.877266+0000 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:59.876 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:40:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:40:59.877 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.877+0000 7ffb2ebfb640 1 Processor -- start 2024-03-25T19:40:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.877+0000 7ffb2ebfb640 1 -- start start 2024-03-25T19:40:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.877+0000 7ffb2ebfb640 10 monclient: init 2024-03-25T19:40:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:40:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb280b4700 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:59.878 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb28142a20 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb280a96b0 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb280a9d60 con 0x7ffb280b4700 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2813ff10 con 0x7ffb28142a20 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb280aa1f0 con 0x7ffb280a96b0 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2ebfb640 10 monclient(hunting): authenticate will time out at 4824.023890s 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2d198640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7ffb280a96b0 0x7ffb2813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.76:32952/0 (socket says 172.21.15.76:32952) 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2c997640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ffb280b4700 0x7ffb28142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52372/0 (socket says 172.21.15.76:52372) 2024-03-25T19:40:59.879 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.878+0000 7ffb2d198640 1 -- 172.21.15.76:0/18628048 learned_addr learned my addr 172.21.15.76:0/18628048 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:40:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059058105 0 0) 0x7ffb2813ff10 con 0x7ffb28142a20 2024-03-25T19:40:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18003520 con 0x7ffb28142a20 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199862329 0 0) 0x7ffb280a9d60 con 0x7ffb280b4700 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2813ff10 con 0x7ffb280b4700 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005654424 0 0) 0x7ffb280aa1f0 con 0x7ffb280a96b0 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:40:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.881+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb280a9d60 con 0x7ffb280a96b0 2024-03-25T19:40:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635810613 0 0) 0x7ffb18003520 con 0x7ffb28142a20 2024-03-25T19:40:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 >> v1:172.21.15.50:6790/0 conn(0x7ffb280a96b0 legacy=0x7ffb2813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 >> v1:172.21.15.50:6789/0 conn(0x7ffb280b4700 legacy=0x7ffb28142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 1 monclient: found mon.noname-c 2024-03-25T19:40:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.883 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb280b4ae0 con 0x7ffb28142a20 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ffb14001690 con 0x7ffb28142a20 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb277fe640 10 monclient: dump: 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.884 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.885 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.885 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.885 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.885 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb2ebfb640 5 monclient: authenticate success, global_id 4126 2024-03-25T19:40:59.885 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.884+0000 7ffb2ebfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb140017f0 con 0x7ffb28142a20 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb277fe640 1 -- 172.21.15.76:0/18628048 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ffb14001c00 con 0x7ffb28142a20 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb2ebfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb2ebfb640 4 set_mon_vals no callback set 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.885+0000 7ffb277fe640 10 monclient: dump: 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.888 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.886+0000 7ffb2ebfb640 10 monclient: shutdown 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.886+0000 7ffb2ebfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.886+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/18628048 >> v1:172.21.15.76:6789/0 conn(0x7ffb28142a20 legacy=0x7ffb2813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.886+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/18628048 shutdown_connections 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.886+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/18628048 >> 172.21.15.76:0/18628048 conn(0x7ffb280668e0 msgr2=0x7ffb28066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:40:59.889 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.886+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/18628048 shutdown_connections 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.886+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/18628048 wait complete. 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 10 monclient: build_initial_monmap 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 10 monclient: monmap: 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.890 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 1 librados: starting msgr at 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 1 librados: starting objecter 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 1 Processor -- start 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 1 -- start start 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 1 librados: setting wanted keys 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 1 librados: calling monclient init 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.887+0000 7ffb2ebfb640 10 monclient: init 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb280a96b0 addr v1:172.21.15.76:6789/0 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb280b4700 addr v1:172.21.15.50:6790/0 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb28142a20 addr v1:172.21.15.50:6789/0 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb281d7500 con 0x7ffb28142a20 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2813cce0 con 0x7ffb280a96b0 2024-03-25T19:40:59.891 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb281d8130 con 0x7ffb280b4700 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2ebfb640 10 monclient(hunting): authenticate will time out at 4824.034158s 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.888+0000 7ffb2c997640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ffb280a96b0 0x7ffb281d7a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:50992/0 (socket says 172.21.15.76:50992) 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.889+0000 7ffb2c997640 1 -- 172.21.15.76:0/1454431867 learned_addr learned my addr 172.21.15.76:0/1454431867 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.889+0000 7ffb257fa640 10 client.?.objecter ms_handle_connect 0x7ffb280a96b0 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.889+0000 7ffb257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.889+0000 7ffb257fa640 10 client.?.objecter ms_handle_connect 0x7ffb28142a20 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.889+0000 7ffb257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.889+0000 7ffb257fa640 10 client.?.objecter ms_handle_connect 0x7ffb280b4700 2024-03-25T19:40:59.892 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.889+0000 7ffb257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:40:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471459662 0 0) 0x7ffb2813cce0 con 0x7ffb280a96b0 2024-03-25T19:40:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.893 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb0c004eb0 con 0x7ffb280a96b0 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837074102 0 0) 0x7ffb281d7500 con 0x7ffb28142a20 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2813cce0 con 0x7ffb28142a20 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822920297 0 0) 0x7ffb281d8130 con 0x7ffb280b4700 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _negotiate 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:40:59.894 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.891+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb281d7500 con 0x7ffb280b4700 2024-03-25T19:40:59.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.896+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 648726698 0 0) 0x7ffb0c004eb0 con 0x7ffb280a96b0 2024-03-25T19:40:59.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.896+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb281d8130 con 0x7ffb280a96b0 2024-03-25T19:40:59.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.896+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ffb1c0019f0 con 0x7ffb280a96b0 2024-03-25T19:40:59.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.896+0000 7ffb257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.896+0000 7ffb257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:40:59.895 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.896+0000 7ffb257fa640 10 monclient(hunting): dump: 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.896 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.896+0000 7ffb257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.897+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1416436151 0 0) 0x7ffb281d7500 con 0x7ffb280b4700 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.897+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb0c004eb0 con 0x7ffb280b4700 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.897+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ffb14001b00 con 0x7ffb280b4700 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.897+0000 7ffb257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.897+0000 7ffb257fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.897+0000 7ffb257fa640 10 monclient(hunting): dump: 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:40:59.897 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:40:59.898 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:40:59.898 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:40:59.898 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:40:59.898 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:40:59.898 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2217895225 0 0) 0x7ffb281d8130 con 0x7ffb280a96b0 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 >> v1:172.21.15.50:6790/0 conn(0x7ffb280b4700 legacy=0x7ffb281d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 >> v1:172.21.15.50:6789/0 conn(0x7ffb28142a20 legacy=0x7ffb281e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 10 monclient: _finish_hunting 0 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 1 monclient: found mon.b 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb281d82c0 con 0x7ffb280a96b0 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 10 monclient: _finish_auth 0 2024-03-25T19:40:59.899 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 10 monclient: _check_auth_tickets 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.899+0000 7ffb257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 10 monclient: _renew_subs 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb281d3fd0 con 0x7ffb280a96b0 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 10 monclient: _renew_subs 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb281d4f40 con 0x7ffb280a96b0 2024-03-25T19:40:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.900+0000 7ffb2ebfb640 1 librados: init done 2024-03-25T19:40:59.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.901+0000 7ffb2ebfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.901+0000 7ffb2ebfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:40:59.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.901+0000 7ffb2ebfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:40:59.901 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:40:59.901+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2804f730 con 0x7ffb280a96b0 2024-03-25T19:41:00.094 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.095+0000 7f4acb62c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:41:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb1c0047e0 con 0x7ffb280a96b0 2024-03-25T19:41:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ffb1c004af0 con 0x7ffb280a96b0 2024-03-25T19:41:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb257fa640 10 monclient: dump: 2024-03-25T19:41:00.105 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffb1c005020 con 0x7ffb280a96b0 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.105+0000 7ffb26ffd640 4 set_mon_vals no callback set 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1318321189 0 0) 0x7ffb1c006600 con 0x7ffb280a96b0 2024-03-25T19:41:00.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 10 client.4129.objecter ms_dispatch 0x7ffb28000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:41:00.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:41:00.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:41:00.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T19:41:00.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ffb1c006a60 con 0x7ffb280a96b0 2024-03-25T19:41:00.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:00.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.106+0000 7ffb257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:00.150 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.151+0000 7fe64582e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:41:00.177 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.178+0000 7ffb2ebfb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ff7c8b95-7815-449d-a015-a11978f86ec7", "id": 2}] 2024-03-25T19:41:00.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.178+0000 7ffb2ebfb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ff7c8b95-7815-449d-a015-a11978f86ec7", "id": 2}] 2024-03-25T19:41:00.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.178+0000 7ffb2ebfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:00.178 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.178+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ff7c8b95-7815-449d-a015-a11978f86ec7", "id": 2} v 0) v1 -- 0x7ffb28069f30 con 0x7ffb280a96b0 2024-03-25T19:41:00.238 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.239+0000 7f4acb62c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:41:00.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.281+0000 7ffb257fa640 1 -- 172.21.15.76:0/1454431867 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff7c8b95-7815-449d-a015-a11978f86ec7", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 685452872 0 216599125) 0x7ffb1c0062b0 con 0x7ffb280a96b0 2024-03-25T19:41:00.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.281+0000 7ffb257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ff7c8b95-7815-449d-a015-a11978f86ec7", "id": 2}] 2024-03-25T19:41:00.280 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.281+0000 7ffb257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:00.281 INFO:teuthology.orchestra.run.smithi076.stdout:2 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 10 librados: watch_flush enter 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 10 librados: watch_flush exit 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 10 client.4129.objecter successfully canceled tick 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 10 monclient: shutdown 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 >> v1:172.21.15.76:6789/0 conn(0x7ffb280a96b0 legacy=0x7ffb281d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 shutdown_connections 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 >> 172.21.15.76:0/1454431867 conn(0x7ffb280668e0 msgr2=0x7ffb2813f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:00.282 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 shutdown_connections 2024-03-25T19:41:00.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 1 -- 172.21.15.76:0/1454431867 wait complete. 2024-03-25T19:41:00.283 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.282+0000 7ffb2ebfb640 1 librados: shutdown 2024-03-25T19:41:00.289 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.291+0000 7fe64582e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:41:00.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new f9bee88b-b66f-48e7-ba74-a053b8686bdd 3 2024-03-25T19:41:00.371 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.372+0000 7f4acb62c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.393+0000 7efd19560640 10 monclient: get_monmap_and_config 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.393+0000 7efd19560640 10 monclient: build_initial_monmap 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.393+0000 7efd19560640 10 monclient: monmap: 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:41:00.393489+0000 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:41:00.393489+0000 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:00.393 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:00.393 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:00.393 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.393+0000 7efd19560640 1 Processor -- start 2024-03-25T19:41:00.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.393+0000 7efd19560640 1 -- start start 2024-03-25T19:41:00.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.393+0000 7efd19560640 10 monclient: init 2024-03-25T19:41:00.394 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 10 monclient(hunting): picked mon.noname-a con 0x7efd140b4700 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 10 monclient(hunting): picked mon.noname-c con 0x7efd14142a20 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 10 monclient(hunting): picked mon.noname-b con 0x7efd140a96b0 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140a9d60 con 0x7efd140b4700 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1413ff10 con 0x7efd14142a20 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140aa1f0 con 0x7efd140a96b0 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd19560640 10 monclient(hunting): authenticate will time out at 4824.540042s 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd137fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7efd140a96b0 0x7efd1413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.76:32966/0 (socket says 172.21.15.76:32966) 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd12ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efd140b4700 0x7efd14142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52384/0 (socket says 172.21.15.76:52384) 2024-03-25T19:41:00.395 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.394+0000 7efd137fe640 1 -- 172.21.15.76:0/2143401866 learned_addr learned my addr 172.21.15.76:0/2143401866 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:00.396 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104913711 0 0) 0x7efd140a9d60 con 0x7efd140b4700 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcfc003520 con 0x7efd140b4700 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949443256 0 0) 0x7efd1413ff10 con 0x7efd14142a20 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd140a9d60 con 0x7efd14142a20 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048370105 0 0) 0x7efd140aa1f0 con 0x7efd140a96b0 2024-03-25T19:41:00.397 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:41:00.398 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.397+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1413ff10 con 0x7efd140a96b0 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 10049977 0 0) 0x7efcfc003520 con 0x7efd140b4700 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 >> v1:172.21.15.50:6790/0 conn(0x7efd140a96b0 legacy=0x7efd1413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 >> v1:172.21.15.76:6789/0 conn(0x7efd14142a20 legacy=0x7efd1413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 1 monclient: found mon.noname-a 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd140b4ae0 con 0x7efd140b4700 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:00.400 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efd04001690 con 0x7efd140b4700 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd19560640 5 monclient: authenticate success, global_id 4134 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd19560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd11ffb640 10 monclient: dump: 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.401 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.402 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.402 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.401+0000 7efd19560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd040017f0 con 0x7efd140b4700 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd11ffb640 1 -- 172.21.15.76:0/2143401866 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efd04001c40 con 0x7efd140b4700 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd11ffb640 10 monclient: dump: 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd19560640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd19560640 4 set_mon_vals no callback set 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd19560640 10 monclient: shutdown 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd19560640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.403+0000 7efd19560640 1 -- 172.21.15.76:0/2143401866 >> v1:172.21.15.50:6789/0 conn(0x7efd140b4700 legacy=0x7efd14142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 1 -- 172.21.15.76:0/2143401866 shutdown_connections 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 1 -- 172.21.15.76:0/2143401866 >> 172.21.15.76:0/2143401866 conn(0x7efd140668e0 msgr2=0x7efd14066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 1 -- 172.21.15.76:0/2143401866 shutdown_connections 2024-03-25T19:41:00.404 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 1 -- 172.21.15.76:0/2143401866 wait complete. 2024-03-25T19:41:00.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 10 monclient: build_initial_monmap 2024-03-25T19:41:00.406 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 10 monclient: monmap: 2024-03-25T19:41:00.406 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:00.406 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 1 librados: starting msgr at 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.404+0000 7efd19560640 1 librados: starting objecter 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 1 Processor -- start 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 1 -- start start 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 1 librados: setting wanted keys 2024-03-25T19:41:00.407 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 1 librados: calling monclient init 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient: init 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient(hunting): picked mon.noname-c con 0x7efd140a96b0 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient(hunting): picked mon.noname-b con 0x7efd140b4700 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient(hunting): picked mon.noname-a con 0x7efd14142a20 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1413b0a0 con 0x7efd14142a20 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1413b340 con 0x7efd140b4700 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1413cc00 con 0x7efd140a96b0 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.405+0000 7efd19560640 10 monclient(hunting): authenticate will time out at 4824.551463s 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efd137fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7efd14142a20 0x7efd141e2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52398/0 (socket says 172.21.15.76:52398) 2024-03-25T19:41:00.408 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efd137fe640 1 -- 172.21.15.76:0/516914125 learned_addr learned my addr 172.21.15.76:0/516914125 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:00.409 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.406+0000 7fe64582e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efcfb7fe640 10 client.?.objecter ms_handle_connect 0x7efd140a96b0 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efcfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efcfb7fe640 10 client.?.objecter ms_handle_connect 0x7efd14142a20 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efcfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efcfb7fe640 10 client.?.objecter ms_handle_connect 0x7efd140b4700 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.406+0000 7efcfb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921970297 0 0) 0x7efd1413cc00 con 0x7efd140a96b0 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.410 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd00004eb0 con 0x7efd140a96b0 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018264651 0 0) 0x7efd1413b0a0 con 0x7efd14142a20 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1413cc00 con 0x7efd14142a20 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411204292 0 0) 0x7efd1413b340 con 0x7efd140b4700 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.408+0000 7efcfb7fe640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:41:00.411 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.409+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1413b0a0 con 0x7efd140b4700 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2507876919 0 0) 0x7efd00004eb0 con 0x7efd140a96b0 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd1413b340 con 0x7efd140a96b0 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efd04001b00 con 0x7efd140a96b0 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.413 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3902033840 0 0) 0x7efd1413cc00 con 0x7efd14142a20 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd00004eb0 con 0x7efd14142a20 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efd0c001bd0 con 0x7efd14142a20 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.414 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.413+0000 7efcfb7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.414+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725031863 0 0) 0x7efd1413b0a0 con 0x7efd140b4700 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.414+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd1413cc00 con 0x7efd140b4700 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.414+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efd080018f0 con 0x7efd140b4700 2024-03-25T19:41:00.415 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.414+0000 7efcfb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.414+0000 7efcfb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.414+0000 7efcfb7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.416 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.417 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1790516103 0 0) 0x7efd1413b340 con 0x7efd140a96b0 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 >> v1:172.21.15.76:6789/0 conn(0x7efd140b4700 legacy=0x7efd141df060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 >> v1:172.21.15.50:6789/0 conn(0x7efd14142a20 legacy=0x7efd141e2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 1 monclient: found mon.c 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1413cd90 con 0x7efd140a96b0 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efcfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 5 monclient: authenticate success, global_id 4139 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 10 monclient: _renew_subs 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:00.418 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1413bf70 con 0x7efd140a96b0 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 10 monclient: _renew_subs 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd14138800 con 0x7efd140a96b0 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.417+0000 7efd19560640 1 librados: init done 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.418+0000 7efd19560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.418+0000 7efd19560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.418+0000 7efd19560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.418+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1404f730 con 0x7efd140a96b0 2024-03-25T19:41:00.420 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.421+0000 7f4acb62c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:41:00.453 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.454+0000 7fe64582e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:41:00.455 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd04001d20 con 0x7efd140a96b0 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efcfb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efd04004cd0 con 0x7efd140a96b0 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efcfb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efcfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efcfb7fe640 10 monclient: dump: 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.456 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.457 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.457 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.457 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efd040051c0 con 0x7efd140a96b0 2024-03-25T19:41:00.457 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.456+0000 7efd117fa640 4 set_mon_vals no callback set 2024-03-25T19:41:00.538 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.539+0000 7f4acb62c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:00.566 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.567+0000 7fe64582e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:00.637 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.638+0000 7f4acb62c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:41:00.643 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2532648930 0 0) 0x7efd04005900 con 0x7efd140a96b0 2024-03-25T19:41:00.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 10 client.4139.objecter ms_dispatch 0x7efd14000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:41:00.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 3 client.4139.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:41:00.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 3 client.4139.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:41:00.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 20 client.4139.objecter dump_active .. 0 homeless 2024-03-25T19:41:00.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efd04005c90 con 0x7efd140a96b0 2024-03-25T19:41:00.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:00.644 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.644+0000 7efcfb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:00.662 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.663+0000 7fe64582e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:41:00.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.718+0000 7efd19560640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f9bee88b-b66f-48e7-ba74-a053b8686bdd", "id": 3}] 2024-03-25T19:41:00.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.718+0000 7efd19560640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f9bee88b-b66f-48e7-ba74-a053b8686bdd", "id": 3}] 2024-03-25T19:41:00.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.718+0000 7efd19560640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:00.717 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.718+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f9bee88b-b66f-48e7-ba74-a053b8686bdd", "id": 3} v 0) v1 -- 0x7efd140b7350 con 0x7efd140a96b0 2024-03-25T19:41:00.802 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.802+0000 7f4acb62c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:00.821 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.822+0000 7fe64582e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:00.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.841+0000 7efcfb7fe640 1 -- 172.21.15.76:0/516914125 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f9bee88b-b66f-48e7-ba74-a053b8686bdd", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2005152354 0 525046306) 0x7efd04005f00 con 0x7efd140a96b0 2024-03-25T19:41:00.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.841+0000 7efcfb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f9bee88b-b66f-48e7-ba74-a053b8686bdd", "id": 3}] 2024-03-25T19:41:00.840 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.841+0000 7efcfb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:00.840 INFO:teuthology.orchestra.run.smithi076.stdout:3 2024-03-25T19:41:00.841 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 10 librados: watch_flush enter 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 10 librados: watch_flush exit 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 20 client.4139.objecter shutdown clearing up homeless session... 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 10 client.4139.objecter successfully canceled tick 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 10 monclient: shutdown 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 >> v1:172.21.15.50:6790/0 conn(0x7efd140a96b0 legacy=0x7efd1413b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 shutdown_connections 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 >> 172.21.15.76:0/516914125 conn(0x7efd140668e0 msgr2=0x7efd140b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 shutdown_connections 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 1 -- 172.21.15.76:0/516914125 wait complete. 2024-03-25T19:41:00.842 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.842+0000 7efd19560640 1 librados: shutdown 2024-03-25T19:41:00.850 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:00.850+0000 7f4acb62c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:41:00.851 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new d0445de8-c8d1-4915-85bf-a033f063e3dc 4 2024-03-25T19:41:00.866 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:00.867+0000 7fe64582e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:41:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.951+0000 7f759057c640 10 monclient: get_monmap_and_config 2024-03-25T19:41:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.951+0000 7f759057c640 10 monclient: build_initial_monmap 2024-03-25T19:41:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.952+0000 7f759057c640 10 monclient: monmap: 2024-03-25T19:41:00.951 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:41:00.952388+0000 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:41:00.952388+0000 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.952+0000 7f759057c640 1 Processor -- start 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.952+0000 7f759057c640 1 -- start start 2024-03-25T19:41:00.952 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.952+0000 7f759057c640 10 monclient: init 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 10 monclient(hunting): picked mon.noname-c con 0x7f75880b4700 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 10 monclient(hunting): picked mon.noname-a con 0x7f7588142a20 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 10 monclient(hunting): picked mon.noname-b con 0x7f75880a96b0 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75880a9d60 con 0x7f7588142a20 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758813ff10 con 0x7f75880b4700 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75880aa1f0 con 0x7f75880a96b0 2024-03-25T19:41:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f759057c640 10 monclient(hunting): authenticate will time out at 4825.099065s 2024-03-25T19:41:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f758db17640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7588142a20 0x7f758813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52414/0 (socket says 172.21.15.76:52414) 2024-03-25T19:41:00.955 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.953+0000 7f758db17640 1 -- 172.21.15.76:0/1725371913 learned_addr learned my addr 172.21.15.76:0/1725371913 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835690029 0 0) 0x7f75880a9d60 con 0x7f7588142a20 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7574003520 con 0x7f7588142a20 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293813084 0 0) 0x7f75880aa1f0 con 0x7f75880a96b0 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.956 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75880a9d60 con 0x7f75880a96b0 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545094666 0 0) 0x7f758813ff10 con 0x7f75880b4700 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:41:00.957 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.956+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75880aa1f0 con 0x7f75880b4700 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 292591894 0 0) 0x7f7574003520 con 0x7f7588142a20 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 >> v1:172.21.15.50:6790/0 conn(0x7f75880a96b0 legacy=0x7f758813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 >> v1:172.21.15.76:6789/0 conn(0x7f75880b4700 legacy=0x7f7588142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 10 monclient: _finish_hunting 0 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 1 monclient: found mon.noname-a 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75880b4ae0 con 0x7f7588142a20 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 10 monclient: _finish_auth 0 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 10 monclient: _check_auth_tickets 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f758d316640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:00.959 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f759057c640 5 monclient: authenticate success, global_id 4140 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.959+0000 7f759057c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.960+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7584001650 con 0x7f7588142a20 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.960+0000 7f758d316640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.960+0000 7f758d316640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.960+0000 7f758d316640 10 monclient: dump: 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.960 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.961 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.961 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.960+0000 7f759057c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:00.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75840018a0 con 0x7f7588142a20 2024-03-25T19:41:00.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f758d316640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:00.963 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f758d316640 1 -- 172.21.15.76:0/1725371913 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7584001c30 con 0x7f7588142a20 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f759057c640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f759057c640 4 set_mon_vals no callback set 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f758d316640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f758d316640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f758d316640 10 monclient: dump: 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.964 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f759057c640 10 monclient: shutdown 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f759057c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.961+0000 7f759057c640 1 -- 172.21.15.76:0/1725371913 >> v1:172.21.15.50:6789/0 conn(0x7f7588142a20 legacy=0x7f758813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 1 -- 172.21.15.76:0/1725371913 shutdown_connections 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 1 -- 172.21.15.76:0/1725371913 >> 172.21.15.76:0/1725371913 conn(0x7f75880668e0 msgr2=0x7f7588066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 1 -- 172.21.15.76:0/1725371913 shutdown_connections 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 1 -- 172.21.15.76:0/1725371913 wait complete. 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 10 monclient: build_initial_monmap 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 10 monclient: monmap: 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:41:00.965 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 1 librados: starting msgr at 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.962+0000 7f759057c640 1 librados: starting objecter 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 1 Processor -- start 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 1 -- start start 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 1 librados: setting wanted keys 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 1 librados: calling monclient init 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient: init 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:00.966 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient(hunting): picked mon.noname-a con 0x7f75880a96b0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient(hunting): picked mon.noname-c con 0x7f75880b4700 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient(hunting): picked mon.noname-b con 0x7f7588142a20 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75880ad150 con 0x7f75880a96b0 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75880ab880 con 0x7f7588142a20 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75880aba10 con 0x7f75880b4700 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.963+0000 7f759057c640 10 monclient(hunting): authenticate will time out at 4825.109321s 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f758e318640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f75880a96b0 0x7f75880aca40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52430/0 (socket says 172.21.15.76:52430) 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f758e318640 1 -- 172.21.15.76:0/1492488056 learned_addr learned my addr 172.21.15.76:0/1492488056 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f757effd640 10 client.?.objecter ms_handle_connect 0x7f75880a96b0 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f757effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f757effd640 10 client.?.objecter ms_handle_connect 0x7f75880b4700 2024-03-25T19:41:00.967 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f757effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:00.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f757effd640 10 client.?.objecter ms_handle_connect 0x7f7588142a20 2024-03-25T19:41:00.968 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.964+0000 7f757effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784890692 0 0) 0x7f75880ad150 con 0x7f75880a96b0 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c004eb0 con 0x7f75880a96b0 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070903775 0 0) 0x7f75880ab880 con 0x7f7588142a20 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.969 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75880ad150 con 0x7f7588142a20 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196962229 0 0) 0x7f75880aba10 con 0x7f75880b4700 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:41:00.970 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.966+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75880ab880 con 0x7f75880b4700 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.970+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2555515352 0 0) 0x7f756c004eb0 con 0x7f75880a96b0 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.970+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75880aba10 con 0x7f75880a96b0 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.970+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7578001a10 con 0x7f75880a96b0 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.970+0000 7f757effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.970+0000 7f757effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.970+0000 7f757effd640 10 monclient(hunting): dump: 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.972 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.970+0000 7f757effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.971+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3971063057 0 0) 0x7f75880ad150 con 0x7f7588142a20 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.971+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f756c004eb0 con 0x7f7588142a20 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.971+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7580001a90 con 0x7f7588142a20 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.971+0000 7f757effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.971+0000 7f757effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.971+0000 7f757effd640 10 monclient(hunting): dump: 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.973 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.972+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 330060186 0 0) 0x7f75880ab880 con 0x7f75880b4700 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.972+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75880ad150 con 0x7f75880b4700 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.972+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7584001a10 con 0x7f75880b4700 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.972+0000 7f757effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.972+0000 7f757effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.972+0000 7f757effd640 10 monclient(hunting): dump: 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.975 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2014135196 0 0) 0x7f75880aba10 con 0x7f75880a96b0 2024-03-25T19:41:00.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 >> v1:172.21.15.50:6790/0 conn(0x7f75880b4700 legacy=0x7f75880ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 >> v1:172.21.15.76:6789/0 conn(0x7f7588142a20 legacy=0x7f75881e2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:00.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:00.977 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 1 monclient: found mon.a 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75880abe50 con 0x7f75880a96b0 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 10 monclient: _finish_auth 0 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f757effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 5 monclient: authenticate success, global_id 4143 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 10 monclient: _renew_subs 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75880aefe0 con 0x7f75880a96b0 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:00.978 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 10 monclient: _renew_subs 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f758813b0e0 con 0x7f75880a96b0 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.974+0000 7f759057c640 1 librados: init done 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.975+0000 7f759057c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.975+0000 7f759057c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.975+0000 7f759057c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.975+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75880ad2e0 con 0x7f75880a96b0 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7578001c70 con 0x7f75880a96b0 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f75780038c0 con 0x7f75880a96b0 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 10 monclient: dump: 2024-03-25T19:41:00.979 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f758cb15640 4 set_mon_vals no callback set 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7578003e70 con 0x7f75880a96b0 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1471522417 0 0) 0x7f75780065f0 con 0x7f75880a96b0 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 10 client.4143.objecter ms_dispatch 0x7f7588000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:41:00.980 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:41:00.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:41:00.981 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:00.978+0000 7f757effd640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T19:41:01.016 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:01.017+0000 7f4acb62c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:41:01.028 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:01.029+0000 7fe64582e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:41:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.079+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f75780061f0 con 0x7f75880a96b0 2024-03-25T19:41:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.079+0000 7f757effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.079+0000 7f757effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:01.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.156+0000 7f759057c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d0445de8-c8d1-4915-85bf-a033f063e3dc", "id": 4}] 2024-03-25T19:41:01.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.156+0000 7f759057c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d0445de8-c8d1-4915-85bf-a033f063e3dc", "id": 4}] 2024-03-25T19:41:01.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.156+0000 7f759057c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:01.155 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.156+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d0445de8-c8d1-4915-85bf-a033f063e3dc", "id": 4} v 0) v1 -- 0x7f75880ac0a0 con 0x7f75880a96b0 2024-03-25T19:41:01.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.200+0000 7f757effd640 1 -- 172.21.15.76:0/1492488056 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d0445de8-c8d1-4915-85bf-a033f063e3dc", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 7196185 0 1697012583) 0x7f7578006410 con 0x7f75880a96b0 2024-03-25T19:41:01.199 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.200+0000 7f757effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d0445de8-c8d1-4915-85bf-a033f063e3dc", "id": 4}] 2024-03-25T19:41:01.200 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.200+0000 7f757effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:01.200 INFO:teuthology.orchestra.run.smithi076.stdout:4 2024-03-25T19:41:01.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 10 librados: watch_flush enter 2024-03-25T19:41:01.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 10 librados: watch_flush exit 2024-03-25T19:41:01.201 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 10 client.4143.objecter successfully canceled tick 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 10 monclient: shutdown 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 >> v1:172.21.15.50:6789/0 conn(0x7f75880a96b0 legacy=0x7f75880aca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 shutdown_connections 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 >> 172.21.15.76:0/1492488056 conn(0x7f75880668e0 msgr2=0x7f75880b6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 shutdown_connections 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 1 -- 172.21.15.76:0/1492488056 wait complete. 2024-03-25T19:41:01.202 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.201+0000 7f759057c640 1 librados: shutdown 2024-03-25T19:41:01.211 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new 115d18c4-4a9f-4476-be16-cb53ccb366b0 5 2024-03-25T19:41:01.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.352+0000 7f761cf2b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:01.351 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.352+0000 7f761cf2b640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.352+0000 7f761cf2b640 10 monclient: monmap: 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:41:01.353129+0000 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:41:01.353129+0000 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:01.352 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.353+0000 7f761cf2b640 1 Processor -- start 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.353+0000 7f761cf2b640 1 -- start start 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.353+0000 7f761cf2b640 10 monclient: init 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.353+0000 7f761cf2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f76180b4700 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f7618142a20 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f76180a96b0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76180a9d60 con 0x7f76180a96b0 2024-03-25T19:41:01.354 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761813ff10 con 0x7f7618142a20 2024-03-25T19:41:01.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76180aa1f0 con 0x7f76180b4700 2024-03-25T19:41:01.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f761cf2b640 10 monclient(hunting): authenticate will time out at 4825.499743s 2024-03-25T19:41:01.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f7615d9b640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7618142a20 0x7f761813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51048/0 (socket says 172.21.15.76:51048) 2024-03-25T19:41:01.355 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.354+0000 7f7615d9b640 1 -- 172.21.15.76:0/1967794739 learned_addr learned my addr 172.21.15.76:0/1967794739 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.356+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496744488 0 0) 0x7f761813ff10 con 0x7f7618142a20 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.356+0000 7f761559a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.356+0000 7f761559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.356+0000 7f761559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.356+0000 7f761559a640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.356+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7600003520 con 0x7f7618142a20 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.357+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196563094 0 0) 0x7f76180aa1f0 con 0x7f76180b4700 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.357+0000 7f761559a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.357+0000 7f761559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.357+0000 7f761559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.356 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.357+0000 7f761559a640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:41:01.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.357+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761813ff10 con 0x7f76180b4700 2024-03-25T19:41:01.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.358+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161993810 0 0) 0x7f76180a9d60 con 0x7f76180a96b0 2024-03-25T19:41:01.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.358+0000 7f761559a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.358+0000 7f761559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.358+0000 7f761559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.357 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.358+0000 7f761559a640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:41:01.358 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.358+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76180aa1f0 con 0x7f76180a96b0 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4142534694 0 0) 0x7f7600003520 con 0x7f7618142a20 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 >> v1:172.21.15.50:6790/0 conn(0x7f76180b4700 legacy=0x7f7618142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 >> v1:172.21.15.50:6789/0 conn(0x7f76180a96b0 legacy=0x7f761813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 10 monclient: _finish_hunting 0 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 1 monclient: found mon.noname-c 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76180b4ae0 con 0x7f7618142a20 2024-03-25T19:41:01.359 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 10 monclient: _finish_auth 0 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 10 monclient: _check_auth_tickets 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761cf2b640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f760c001690 con 0x7f7618142a20 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761cf2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761559a640 10 monclient: dump: 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.360 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.361 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.361 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.361 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.361 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.360+0000 7f761cf2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:01.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f760c0017f0 con 0x7f7618142a20 2024-03-25T19:41:01.362 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761559a640 1 -- 172.21.15.76:0/1967794739 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f760c001c00 con 0x7f7618142a20 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761cf2b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761cf2b640 4 set_mon_vals no callback set 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.361+0000 7f761559a640 10 monclient: dump: 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.363 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.364 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.364 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.362+0000 7f761cf2b640 10 monclient: shutdown 2024-03-25T19:41:01.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.362+0000 7f761cf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:01.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.362+0000 7f761cf2b640 1 -- 172.21.15.76:0/1967794739 >> v1:172.21.15.76:6789/0 conn(0x7f7618142a20 legacy=0x7f761813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.362+0000 7f761cf2b640 1 -- 172.21.15.76:0/1967794739 shutdown_connections 2024-03-25T19:41:01.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.362+0000 7f761cf2b640 1 -- 172.21.15.76:0/1967794739 >> 172.21.15.76:0/1967794739 conn(0x7f76180668e0 msgr2=0x7f7618066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:01.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.362+0000 7f761cf2b640 1 -- 172.21.15.76:0/1967794739 shutdown_connections 2024-03-25T19:41:01.365 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.362+0000 7f761cf2b640 1 -- 172.21.15.76:0/1967794739 wait complete. 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 10 monclient: monmap: 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 1 librados: starting msgr at 2024-03-25T19:41:01.366 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 1 librados: starting objecter 2024-03-25T19:41:01.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 1 Processor -- start 2024-03-25T19:41:01.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 1 -- start start 2024-03-25T19:41:01.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 1 librados: setting wanted keys 2024-03-25T19:41:01.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 1 librados: calling monclient init 2024-03-25T19:41:01.367 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.363+0000 7f761cf2b640 10 monclient: init 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 10 monclient(hunting): picked mon.noname-a con 0x7f76180a96b0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 10 monclient(hunting): picked mon.noname-b con 0x7f76180b4700 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 10 monclient(hunting): picked mon.noname-c con 0x7f7618142a20 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76181d7600 con 0x7f76180a96b0 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76181d3fd0 con 0x7f76180b4700 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76181d4160 con 0x7f7618142a20 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.369 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.364+0000 7f761cf2b640 10 monclient(hunting): authenticate will time out at 4825.510186s 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.365+0000 7f7615d9b640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f76180b4700 0x7f761813b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51052/0 (socket says 172.21.15.76:51052) 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.365+0000 7f7615d9b640 1 -- 172.21.15.76:0/3240489157 learned_addr learned my addr 172.21.15.76:0/3240489157 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.365+0000 7f7606ffd640 10 client.?.objecter ms_handle_connect 0x7f76180b4700 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.365+0000 7f7606ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.365+0000 7f7606ffd640 10 client.?.objecter ms_handle_connect 0x7f76180a96b0 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.365+0000 7f7606ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954841256 0 0) 0x7f76181d3fd0 con 0x7f76180b4700 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:41:01.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f8004eb0 con 0x7f76180b4700 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 10 client.?.objecter ms_handle_connect 0x7f7618142a20 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.367+0000 7f7606ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.368+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036107311 0 0) 0x7f76181d7600 con 0x7f76180a96b0 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.368+0000 7f7606ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.368+0000 7f7606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.368+0000 7f7606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.368+0000 7f7606ffd640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.368+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76181d3fd0 con 0x7f76180a96b0 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.369+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991351395 0 0) 0x7f76181d4160 con 0x7f7618142a20 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.369+0000 7f7606ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.369+0000 7f7606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.369+0000 7f7606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.369+0000 7f7606ffd640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:41:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.369+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76181d7600 con 0x7f7618142a20 2024-03-25T19:41:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.371+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 132623555 0 0) 0x7f75f8004eb0 con 0x7f76180b4700 2024-03-25T19:41:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.371+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76181d4160 con 0x7f76180b4700 2024-03-25T19:41:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.371+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f760c001bc0 con 0x7f76180b4700 2024-03-25T19:41:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.371+0000 7f7606ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.372 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.371+0000 7f7606ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.371+0000 7f7606ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.371+0000 7f7606ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:01.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 492116879 0 0) 0x7f76181d7600 con 0x7f7618142a20 2024-03-25T19:41:01.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75f8004eb0 con 0x7f7618142a20 2024-03-25T19:41:01.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7610001c70 con 0x7f7618142a20 2024-03-25T19:41:01.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:01.375 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3223769940 0 0) 0x7f76181d4160 con 0x7f76180b4700 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 >> v1:172.21.15.50:6790/0 conn(0x7f7618142a20 legacy=0x7f76181d4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 >> v1:172.21.15.50:6789/0 conn(0x7f76180a96b0 legacy=0x7f76181d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.376 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 monclient: found mon.b 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761813cb80 con 0x7f76180b4700 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f7606ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 10 monclient: _renew_subs 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76181d81a0 con 0x7f76180b4700 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 10 monclient: _renew_subs 2024-03-25T19:41:01.377 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:01.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f761813bd40 con 0x7f76180b4700 2024-03-25T19:41:01.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.374+0000 7f761cf2b640 1 librados: init done 2024-03-25T19:41:01.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.375+0000 7f761cf2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.375+0000 7f761cf2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.375+0000 7f761cf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:01.378 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.375+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761804f730 con 0x7f76180b4700 2024-03-25T19:41:01.471 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f760c0011f0 con 0x7f76180b4700 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f760c004c30 con 0x7f76180b4700 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 10 monclient: dump: 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.472 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f760c005160 con 0x7f76180b4700 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 4222141476 0 0) 0x7f760c0068c0 con 0x7f76180b4700 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7614d99640 4 set_mon_vals no callback set 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 10 client.4147.objecter ms_dispatch 0x7f7618000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f760c018dc0 con 0x7f76180b4700 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.473 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.472+0000 7f7606ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:01.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.544+0000 7f761cf2b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "115d18c4-4a9f-4476-be16-cb53ccb366b0", "id": 5}] 2024-03-25T19:41:01.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.544+0000 7f761cf2b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "115d18c4-4a9f-4476-be16-cb53ccb366b0", "id": 5}] 2024-03-25T19:41:01.543 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.544+0000 7f761cf2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:01.544 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.544+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd new", "uuid": "115d18c4-4a9f-4476-be16-cb53ccb366b0", "id": 5} v 0) v1 -- 0x7f7618069f30 con 0x7f76180b4700 2024-03-25T19:41:01.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.626+0000 7f7606ffd640 1 -- 172.21.15.76:0/3240489157 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "115d18c4-4a9f-4476-be16-cb53ccb366b0", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 4142627295 0 1988412176) 0x7f760c0063d0 con 0x7f76180b4700 2024-03-25T19:41:01.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.626+0000 7f7606ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "115d18c4-4a9f-4476-be16-cb53ccb366b0", "id": 5}] 2024-03-25T19:41:01.625 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.626+0000 7f7606ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:01.625 INFO:teuthology.orchestra.run.smithi076.stdout:5 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 10 librados: watch_flush enter 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 10 librados: watch_flush exit 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 10 client.4147.objecter successfully canceled tick 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 10 monclient: shutdown 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 >> v1:172.21.15.76:6789/0 conn(0x7f76180b4700 legacy=0x7f761813b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 shutdown_connections 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 >> 172.21.15.76:0/3240489157 conn(0x7f76180668e0 msgr2=0x7f761813da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 shutdown_connections 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 1 -- 172.21.15.76:0/3240489157 wait complete. 2024-03-25T19:41:01.627 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.627+0000 7f761cf2b640 1 librados: shutdown 2024-03-25T19:41:01.636 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new 7cc327fd-d9f4-4632-a4a5-551ac2414a09 6 2024-03-25T19:41:01.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.738+0000 7f583fb24640 10 monclient: get_monmap_and_config 2024-03-25T19:41:01.737 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.738+0000 7f583fb24640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.738+0000 7f583fb24640 10 monclient: monmap: 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:41:01.739206+0000 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:41:01.739206+0000 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:01.738 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.739+0000 7f583fb24640 1 Processor -- start 2024-03-25T19:41:01.739 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.739+0000 7f583fb24640 1 -- start start 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.739+0000 7f583fb24640 10 monclient: init 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 10 monclient(hunting): picked mon.noname-a con 0x7f5838070210 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 10 monclient(hunting): picked mon.noname-c con 0x7f58380b06d0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 10 monclient(hunting): picked mon.noname-b con 0x7f58380705f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58380b0b80 con 0x7f5838070210 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58380b0430 con 0x7f58380b06d0 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5838071670 con 0x7f58380705f0 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583fb24640 10 monclient(hunting): authenticate will time out at 4825.885948s 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583d8c0640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5838070210 0x7f5838140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52464/0 (socket says 172.21.15.76:52464) 2024-03-25T19:41:01.740 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.740+0000 7f583d8c0640 1 -- 172.21.15.76:0/369373155 learned_addr learned my addr 172.21.15.76:0/369373155 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267047690 0 0) 0x7f5834001110 con 0x7f58380b06d0 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5824003520 con 0x7f58380b06d0 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822025632 0 0) 0x7f58380b0b80 con 0x7f5838070210 2024-03-25T19:41:01.742 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:41:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.743+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5834001110 con 0x7f5838070210 2024-03-25T19:41:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.744+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474802567 0 0) 0x7f5838071670 con 0x7f58380705f0 2024-03-25T19:41:01.743 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.744+0000 7f583c8be640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.744+0000 7f583c8be640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.744+0000 7f583c8be640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.744+0000 7f583c8be640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:41:01.744 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.744+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58380b0b80 con 0x7f58380705f0 2024-03-25T19:41:01.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3619815188 0 0) 0x7f5834001110 con 0x7f5838070210 2024-03-25T19:41:01.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 >> v1:172.21.15.50:6790/0 conn(0x7f58380705f0 legacy=0x7f583814b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 >> v1:172.21.15.76:6789/0 conn(0x7f58380b06d0 legacy=0x7f58380afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 10 monclient: _finish_hunting 0 2024-03-25T19:41:01.746 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 1 monclient: found mon.noname-a 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838070a60 con 0x7f5838070210 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 10 monclient: _finish_auth 0 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 10 monclient: _check_auth_tickets 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5828001670 con 0x7f5838070210 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583fb24640 5 monclient: authenticate success, global_id 4152 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583fb24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583c8be640 10 monclient: dump: 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.747 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.748 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.747+0000 7f583fb24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58280017d0 con 0x7f5838070210 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583c8be640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583c8be640 1 -- 172.21.15.76:0/369373155 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5828001b60 con 0x7f5838070210 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583fb24640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583fb24640 4 set_mon_vals no callback set 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583c8be640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583c8be640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.748+0000 7f583c8be640 10 monclient: dump: 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.749+0000 7f583fb24640 10 monclient: shutdown 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.749+0000 7f583fb24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.749+0000 7f583fb24640 1 -- 172.21.15.76:0/369373155 >> v1:172.21.15.50:6789/0 conn(0x7f5838070210 legacy=0x7f5838140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.749+0000 7f583fb24640 1 -- 172.21.15.76:0/369373155 shutdown_connections 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.749+0000 7f583fb24640 1 -- 172.21.15.76:0/369373155 >> 172.21.15.76:0/369373155 conn(0x7f58380b5cd0 msgr2=0x7f58380b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.749+0000 7f583fb24640 1 -- 172.21.15.76:0/369373155 shutdown_connections 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.749+0000 7f583fb24640 1 -- 172.21.15.76:0/369373155 wait complete. 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 10 monclient: build_initial_monmap 2024-03-25T19:41:01.752 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 10 monclient: monmap: 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 1 librados: starting msgr at 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 1 librados: starting objecter 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 1 Processor -- start 2024-03-25T19:41:01.753 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 1 -- start start 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 1 librados: setting wanted keys 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 1 librados: calling monclient init 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.750+0000 7f583fb24640 10 monclient: init 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 10 monclient(hunting): picked mon.noname-b con 0x7f5838070210 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 10 monclient(hunting): picked mon.noname-c con 0x7f58380705f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 10 monclient(hunting): picked mon.noname-a con 0x7f58380b06d0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583806c870 con 0x7f58380b06d0 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583806cb50 con 0x7f5838070210 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583806ce30 con 0x7f58380705f0 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:01.754 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583fb24640 10 monclient(hunting): authenticate will time out at 4825.896954s 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583e0c1640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f58380b06d0 0x7f58381312f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52474/0 (socket says 172.21.15.76:52474) 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f583e0c1640 1 -- 172.21.15.76:0/2658408077 learned_addr learned my addr 172.21.15.76:0/2658408077 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f582e7fc640 10 client.?.objecter ms_handle_connect 0x7f58380b06d0 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.751+0000 7f582e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.752+0000 7f582e7fc640 10 client.?.objecter ms_handle_connect 0x7f5838070210 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.752+0000 7f582e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.752+0000 7f582e7fc640 10 client.?.objecter ms_handle_connect 0x7f58380705f0 2024-03-25T19:41:01.755 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.752+0000 7f582e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.753+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267399687 0 0) 0x7f583806c870 con 0x7f58380b06d0 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.753+0000 7f582e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.753+0000 7f582e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.753+0000 7f582e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.753+0000 7f582e7fc640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.753+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f581c004eb0 con 0x7f58380b06d0 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804165861 0 0) 0x7f583806cb50 con 0x7f5838070210 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.756 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f583806c870 con 0x7f5838070210 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555210741 0 0) 0x7f583806ce30 con 0x7f58380705f0 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:41:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.754+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f583806cb50 con 0x7f58380705f0 2024-03-25T19:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.759+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3886471568 0 0) 0x7f581c004eb0 con 0x7f58380b06d0 2024-03-25T19:41:01.758 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.759+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f583806ce30 con 0x7f58380b06d0 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.759+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5830001d10 con 0x7f58380b06d0 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.759+0000 7f582e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.759+0000 7f582e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.759+0000 7f582e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.759 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.760 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.759+0000 7f582e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:01.760 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3063354660 0 0) 0x7f583806c870 con 0x7f5838070210 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f581c004eb0 con 0x7f5838070210 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5828001ae0 con 0x7f5838070210 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.761 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1709851759 0 0) 0x7f583806cb50 con 0x7f58380705f0 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f583806c870 con 0x7f58380705f0 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5834001570 con 0x7f58380705f0 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.761+0000 7f582e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.762 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.763 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1813129631 0 0) 0x7f583806ce30 con 0x7f58380b06d0 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 >> v1:172.21.15.50:6790/0 conn(0x7f58380705f0 legacy=0x7f5838130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 >> v1:172.21.15.76:6789/0 conn(0x7f5838070210 legacy=0x7f583813bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 1 monclient: found mon.a 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58380ae7d0 con 0x7f58380b06d0 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f582e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 5 monclient: authenticate success, global_id 4155 2024-03-25T19:41:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 10 monclient: _renew_subs 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583806bcb0 con 0x7f58380b06d0 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 10 monclient: _renew_subs 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.762+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f583813c1e0 con 0x7f58380b06d0 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.763+0000 7f583fb24640 1 librados: init done 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.763+0000 7f583fb24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.763+0000 7f583fb24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.763+0000 7f583fb24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.763+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583804f730 con 0x7f58380b06d0 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5830003990 con 0x7f58380b06d0 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5830003d60 con 0x7f58380b06d0 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582e7fc640 10 monclient: dump: 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:01.867 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5830004310 con 0x7f58380b06d0 2024-03-25T19:41:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.868+0000 7f582ffff640 4 set_mon_vals no callback set 2024-03-25T19:41:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1761854980 0 0) 0x7f5830006c10 con 0x7f58380b06d0 2024-03-25T19:41:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 10 client.4155.objecter ms_dispatch 0x7f5838000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:41:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:41:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:41:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T19:41:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5830006620 con 0x7f58380b06d0 2024-03-25T19:41:01.869 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:01.870 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.869+0000 7f582e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:01.874 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:01.876+0000 7fe64582e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:41:01.883 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:01.884+0000 7f4acb62c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:41:01.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.941+0000 7f583fb24640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7cc327fd-d9f4-4632-a4a5-551ac2414a09", "id": 6}] 2024-03-25T19:41:01.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.941+0000 7f583fb24640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7cc327fd-d9f4-4632-a4a5-551ac2414a09", "id": 6}] 2024-03-25T19:41:01.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.941+0000 7f583fb24640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:01.940 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.941+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7cc327fd-d9f4-4632-a4a5-551ac2414a09", "id": 6} v 0) v1 -- 0x7f5838069f30 con 0x7f58380b06d0 2024-03-25T19:41:01.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.972+0000 7f582e7fc640 1 -- 172.21.15.76:0/2658408077 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7cc327fd-d9f4-4632-a4a5-551ac2414a09", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4052291510 0 1113811849) 0x7f5830006840 con 0x7f58380b06d0 2024-03-25T19:41:01.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.972+0000 7f582e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7cc327fd-d9f4-4632-a4a5-551ac2414a09", "id": 6}] 2024-03-25T19:41:01.971 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.972+0000 7f582e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:01.971 INFO:teuthology.orchestra.run.smithi076.stdout:6 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 10 librados: watch_flush enter 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 10 librados: watch_flush exit 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 10 client.4155.objecter successfully canceled tick 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 10 monclient: shutdown 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 >> v1:172.21.15.50:6789/0 conn(0x7f58380b06d0 legacy=0x7f58381312f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 shutdown_connections 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 >> 172.21.15.76:0/2658408077 conn(0x7f58380b5cd0 msgr2=0x7f58380b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:01.973 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 shutdown_connections 2024-03-25T19:41:01.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 1 -- 172.21.15.76:0/2658408077 wait complete. 2024-03-25T19:41:01.974 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:01.973+0000 7f583fb24640 1 librados: shutdown 2024-03-25T19:41:01.982 DEBUG:teuthology.orchestra.run.smithi076:> sudo ceph --cluster ceph osd new f4c445e7-27c7-4718-a82d-16a303cb47b4 7 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.083+0000 7f8fdeee6640 10 monclient: get_monmap_and_config 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.083+0000 7f8fdeee6640 10 monclient: build_initial_monmap 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.083+0000 7f8fdeee6640 10 monclient: monmap: 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:41:02.083606+0000 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:41:02.083606+0000 2024-03-25T19:41:02.082 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:02.083 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:02.083 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:02.083 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:02.083 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:02.083 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.083+0000 7f8fdeee6640 1 Processor -- start 2024-03-25T19:41:02.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.083+0000 7f8fdeee6640 1 -- start start 2024-03-25T19:41:02.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.083+0000 7f8fdeee6640 10 monclient: init 2024-03-25T19:41:02.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:02.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:02.084 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd80a9920 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd80b45a0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd8146450 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd806b100 con 0x7f8fd80a9920 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80b2940 con 0x7f8fd80b45a0 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd813fec0 con 0x7f8fd8146450 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.084+0000 7f8fdeee6640 10 monclient(hunting): authenticate will time out at 4826.230246s 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.085+0000 7f8fdcc82640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f8fd80a9920 0x7f8fd813f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52488/0 (socket says 172.21.15.76:52488) 2024-03-25T19:41:02.085 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.085+0000 7f8fdcc82640 1 -- 172.21.15.76:0/3508149756 learned_addr learned my addr 172.21.15.76:0/3508149756 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910295506 0 0) 0x7f8fd80b2940 con 0x7f8fd80b45a0 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc8003520 con 0x7f8fd80b45a0 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227928976 0 0) 0x7f8fd806b100 con 0x7f8fd80a9920 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd80b2940 con 0x7f8fd80a9920 2024-03-25T19:41:02.087 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273704722 0 0) 0x7f8fd813fec0 con 0x7f8fd8146450 2024-03-25T19:41:02.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:41:02.088 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.087+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd806b100 con 0x7f8fd8146450 2024-03-25T19:41:02.090 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3758786928 0 0) 0x7f8fd80b2940 con 0x7f8fd80a9920 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 >> v1:172.21.15.50:6790/0 conn(0x7f8fd8146450 legacy=0x7f8fd814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 >> v1:172.21.15.76:6789/0 conn(0x7f8fd80b45a0 legacy=0x7f8fd8145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 1 monclient: found mon.noname-a 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd8140050 con 0x7f8fd80a9920 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8fcc001670 con 0x7f8fd80a9920 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:02.091 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fd77fe640 10 monclient: dump: 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fdeee6640 5 monclient: authenticate success, global_id 4158 2024-03-25T19:41:02.092 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.091+0000 7f8fdeee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:02.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fcc0017d0 con 0x7f8fd80a9920 2024-03-25T19:41:02.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fd77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:02.095 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fd77fe640 1 -- 172.21.15.76:0/3508149756 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8fcc001a40 con 0x7f8fd80a9920 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 4 set_mon_vals no callback set 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fd77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fd77fe640 10 monclient: dump: 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.096 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 10 monclient: shutdown 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 1 -- 172.21.15.76:0/3508149756 >> v1:172.21.15.50:6789/0 conn(0x7f8fd80a9920 legacy=0x7f8fd813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 1 -- 172.21.15.76:0/3508149756 shutdown_connections 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 1 -- 172.21.15.76:0/3508149756 >> 172.21.15.76:0/3508149756 conn(0x7f8fd80b7f40 msgr2=0x7f8fd80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 1 -- 172.21.15.76:0/3508149756 shutdown_connections 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.093+0000 7f8fdeee6640 1 -- 172.21.15.76:0/3508149756 wait complete. 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 10 monclient: build_initial_monmap 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 10 monclient: monmap: 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 0 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 0.000000 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:created 0.000000 2024-03-25T19:41:02.097 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 1 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 1 librados: starting msgr at 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 1 librados: starting objecter 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 1 Processor -- start 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 1 -- start start 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 1 librados: setting wanted keys 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 1 librados: calling monclient init 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.094+0000 7f8fdeee6640 10 monclient: init 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:02.098 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd80a9920 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd80b45a0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd8146450 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd8135a90 con 0x7f8fd80b45a0 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd8145660 con 0x7f8fd8146450 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd8144620 con 0x7f8fd80a9920 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.095+0000 7f8fdeee6640 10 monclient(hunting): authenticate will time out at 4826.241252s 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fdd483640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8fd8146450 0x7f8fd8135380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:51100/0 (socket says 172.21.15.76:51100) 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fdd483640 1 -- 172.21.15.76:0/154309408 learned_addr learned my addr 172.21.15.76:0/154309408 (peer_addr_for_me v1:172.21.15.76:0/0) 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fd7fff640 1 --1- 172.21.15.76:0/154309408 >> v1:172.21.15.50:6789/0 conn(0x7f8fd80b45a0 0x7f8fd8134c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.76:52504/0 (socket says 172.21.15.76:52504) 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fd57fa640 10 client.?.objecter ms_handle_connect 0x7f8fd8146450 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fd57fa640 10 client.?.objecter ms_handle_connect 0x7f8fd80b45a0 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fd57fa640 10 client.?.objecter ms_handle_connect 0x7f8fd80a9920 2024-03-25T19:41:02.099 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.096+0000 7f8fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.100 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.098+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241510146 0 0) 0x7f8fd8145660 con 0x7f8fd8146450 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.098+0000 7f8fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.098+0000 7f8fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.098+0000 7f8fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.098+0000 7f8fd57fa640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.098+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fbc004eb0 con 0x7f8fd8146450 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.099+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651628918 0 0) 0x7f8fd8144620 con 0x7f8fd80a9920 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.099+0000 7f8fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.099+0000 7f8fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.099+0000 7f8fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.099+0000 7f8fd57fa640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.099+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd8145660 con 0x7f8fd80a9920 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.100+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638656161 0 0) 0x7f8fd8135a90 con 0x7f8fd80b45a0 2024-03-25T19:41:02.101 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.100+0000 7f8fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.100+0000 7f8fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.100+0000 7f8fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.100+0000 7f8fd57fa640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:41:02.102 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.100+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd8144620 con 0x7f8fd80b45a0 2024-03-25T19:41:02.103 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1748350169 0 0) 0x7f8fd8145660 con 0x7f8fd80a9920 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd8135a90 con 0x7f8fd80a9920 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8fcc001940 con 0x7f8fd80a9920 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 10 monclient(hunting): dump: 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.104 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215909221 0 0) 0x7f8fd8144620 con 0x7f8fd80b45a0 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd8145660 con 0x7f8fd80b45a0 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8fc4001950 con 0x7f8fd80b45a0 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.104+0000 7f8fd57fa640 10 monclient(hunting): dump: 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:02.105 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.106 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.106 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.106 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.105+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1532710815 0 0) 0x7f8fbc004eb0 con 0x7f8fd8146450 2024-03-25T19:41:02.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.105+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd8144620 con 0x7f8fd8146450 2024-03-25T19:41:02.106 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.105+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8fd0001c70 con 0x7f8fd8146450 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.105+0000 7f8fd57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.105+0000 7f8fd57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.105+0000 7f8fd57fa640 10 monclient(hunting): dump: 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.107 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2376960377 0 0) 0x7f8fd8135a90 con 0x7f8fd80a9920 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 >> v1:172.21.15.76:6789/0 conn(0x7f8fd8146450 legacy=0x7f8fd8135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 >> v1:172.21.15.50:6789/0 conn(0x7f8fd80b45a0 legacy=0x7f8fd8134c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 1 monclient: found mon.c 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd81424a0 con 0x7f8fd80a9920 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fd57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 5 monclient: authenticate success, global_id 4163 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 10 monclient: _renew_subs 2024-03-25T19:41:02.109 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd81426b0 con 0x7f8fd80a9920 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 10 monclient: _renew_subs 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd813c680 con 0x7f8fd80a9920 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.108+0000 7f8fdeee6640 1 librados: init done 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.109+0000 7f8fdeee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.109+0000 7f8fdeee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.109+0000 7f8fdeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.110 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.109+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd804f730 con 0x7f8fd80a9920 2024-03-25T19:41:02.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fcc004a90 con 0x7f8fd80a9920 2024-03-25T19:41:02.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:02.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8fcc004da0 con 0x7f8fd80a9920 2024-03-25T19:41:02.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:02.112 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 10 monclient: dump: 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:epoch 1 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:election_strategy: 3 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8fcc005290 con 0x7f8fd80a9920 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd6ffd640 4 set_mon_vals no callback set 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 329964656 0 0) 0x7f8fcc006910 con 0x7f8fd80a9920 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 10 client.4163.objecter ms_dispatch 0x7f8fd8000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 3 client.4163.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:41:02.113 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 3 client.4163.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:41:02.114 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.113+0000 7f8fd57fa640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-25T19:41:02.171 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:02.172+0000 7fe64582e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:41:02.181 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:02.181+0000 7f4acb62c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:41:02.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.220+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8fcc006b20 con 0x7f8fd80a9920 2024-03-25T19:41:02.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.220+0000 7f8fd57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.219 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.220+0000 7f8fd57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:02.222 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:02.223+0000 7fe64582e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:41:02.241 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:02.242+0000 7f4acb62c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:41:02.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.297+0000 7f8fdeee6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f4c445e7-27c7-4718-a82d-16a303cb47b4", "id": 7}] 2024-03-25T19:41:02.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.297+0000 7f8fdeee6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f4c445e7-27c7-4718-a82d-16a303cb47b4", "id": 7}] 2024-03-25T19:41:02.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.297+0000 7f8fdeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.296 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.297+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f4c445e7-27c7-4718-a82d-16a303cb47b4", "id": 7} v 0) v1 -- 0x7f8fd8145260 con 0x7f8fd80a9920 2024-03-25T19:41:02.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.369+0000 7f8fd57fa640 1 -- 172.21.15.76:0/154309408 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4c445e7-27c7-4718-a82d-16a303cb47b4", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 4023367784 0 1371665406) 0x7f8fcc006360 con 0x7f8fd80a9920 2024-03-25T19:41:02.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.369+0000 7f8fd57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f4c445e7-27c7-4718-a82d-16a303cb47b4", "id": 7}] 2024-03-25T19:41:02.368 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.369+0000 7f8fd57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:02.368 INFO:teuthology.orchestra.run.smithi076.stdout:7 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.369+0000 7f8fdeee6640 10 librados: watch_flush enter 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 10 librados: watch_flush exit 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 10 client.4163.objecter successfully canceled tick 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 10 monclient: shutdown 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 >> v1:172.21.15.50:6790/0 conn(0x7f8fd80a9920 legacy=0x7f8fd81428d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 shutdown_connections 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 >> 172.21.15.76:0/154309408 conn(0x7f8fd80b7f40 msgr2=0x7f8fd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:02.370 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 shutdown_connections 2024-03-25T19:41:02.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 1 -- 172.21.15.76:0/154309408 wait complete. 2024-03-25T19:41:02.371 INFO:teuthology.orchestra.run.smithi076.stderr:2024-03-25T19:41:02.370+0000 7f8fdeee6640 1 librados: shutdown 2024-03-25T19:41:02.371 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:02.372+0000 7fe64582e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:41:02.380 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.380 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:41:02.380 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:41:02.381 INFO:tasks.ceph.mgr.y.smithi050.stderr:2024-03-25T19:41:02.382+0000 7f4acb62c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:41:02.382 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:41:02.383 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.383 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:41:02.383 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:41:02.385 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:41:02.385 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.385 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:41:02.385 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:41:02.387 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:41:02.387 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.388 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:41:02.388 DEBUG:teuthology.orchestra.run.smithi050:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:41:02.390 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:41:02.390 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.390 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:41:02.390 DEBUG:teuthology.orchestra.run.smithi076:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:41:02.392 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:41:02.393 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.393 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:41:02.393 DEBUG:teuthology.orchestra.run.smithi076:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:41:02.395 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:41:02.395 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.395 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:41:02.395 DEBUG:teuthology.orchestra.run.smithi076:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:41:02.397 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:41:02.397 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:02.397 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:41:02.397 DEBUG:teuthology.orchestra.run.smithi076:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:41:02.399 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:41:02.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:02.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.555+0000 7f718f3ba640 10 monclient: get_monmap_and_config 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.555+0000 7f718f3ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.555+0000 7f718f3ba640 10 monclient: monmap: 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:41:02.555999+0000 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:41:02.555999+0000 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:02.555 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:02.556 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:41:02.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.556+0000 7f718f3ba640 1 Processor -- start 2024-03-25T19:41:02.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.556+0000 7f718f3ba640 1 -- start start 2024-03-25T19:41:02.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.556+0000 7f718f3ba640 10 monclient: init 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 10 monclient(hunting): picked mon.noname-b con 0x7f71880a9920 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 10 monclient(hunting): picked mon.noname-c con 0x7f71880b45a0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 10 monclient(hunting): picked mon.noname-a con 0x7f7188146450 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718806b100 con 0x7f7188146450 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71880b2940 con 0x7f71880b45a0 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718813fec0 con 0x7f71880a9920 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.557+0000 7f718f3ba640 10 monclient(hunting): authenticate will time out at 4837.820983s 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.558+0000 7f718d156640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f71880a9920 0x7f718813f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:40828/0 (socket says 172.21.15.50:40828) 2024-03-25T19:41:02.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.558+0000 7f718d156640 1 -- 172.21.15.50:0/3719524715 learned_addr learned my addr 172.21.15.50:0/3719524715 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:02.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.733+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517857818 0 0) 0x7f71880b2940 con 0x7f71880b45a0 2024-03-25T19:41:02.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.734+0000 7f717ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.734+0000 7f717ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.734+0000 7f717ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.734+0000 7f717ffff640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:41:02.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.734+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7174003520 con 0x7f71880b45a0 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 16658114 0 0) 0x7f7174003520 con 0x7f71880b45a0 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 >> v1:172.21.15.50:6790/0 conn(0x7f71880a9920 legacy=0x7f718813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 >> v1:172.21.15.50:6789/0 conn(0x7f7188146450 legacy=0x7f718814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 1 monclient: found mon.noname-c 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7188140050 con 0x7f71880b45a0 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 10 monclient: _finish_auth 0 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:02.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7178001690 con 0x7f71880b45a0 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f717ffff640 10 monclient: dump: 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f718f3ba640 5 monclient: authenticate success, global_id 4162 2024-03-25T19:41:02.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.752+0000 7f718f3ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71780017f0 con 0x7f71880b45a0 2024-03-25T19:41:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f717ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f717ffff640 1 -- 172.21.15.50:0/3719524715 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7178001aa0 con 0x7f71880b45a0 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f718f3ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f718f3ba640 4 set_mon_vals no callback set 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f717ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f717ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.974+0000 7f717ffff640 10 monclient: dump: 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.974 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.975 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.975 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:41:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.975+0000 7f718f3ba640 10 monclient: shutdown 2024-03-25T19:41:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.975+0000 7f718f3ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.975+0000 7f718f3ba640 1 -- 172.21.15.50:0/3719524715 >> v1:172.21.15.76:6789/0 conn(0x7f71880b45a0 legacy=0x7f7188145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.975+0000 7f718f3ba640 1 -- 172.21.15.50:0/3719524715 shutdown_connections 2024-03-25T19:41:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.975+0000 7f718f3ba640 1 -- 172.21.15.50:0/3719524715 >> 172.21.15.50:0/3719524715 conn(0x7f71880b7f40 msgr2=0x7f71880b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:02.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.975+0000 7f718f3ba640 1 -- 172.21.15.50:0/3719524715 shutdown_connections 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.975+0000 7f718f3ba640 1 -- 172.21.15.50:0/3719524715 wait complete. 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.976+0000 7f718f3ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.976+0000 7f718f3ba640 10 monclient: monmap: 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 0 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:created 0.000000 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.976+0000 7f718f3ba640 1 librados: starting msgr at 2024-03-25T19:41:02.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.976+0000 7f718f3ba640 1 librados: starting objecter 2024-03-25T19:41:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 1 Processor -- start 2024-03-25T19:41:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 1 -- start start 2024-03-25T19:41:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 1 librados: setting wanted keys 2024-03-25T19:41:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 1 librados: calling monclient init 2024-03-25T19:41:02.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 10 monclient: init 2024-03-25T19:41:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 10 monclient(hunting): picked mon.noname-c con 0x7f71880a9920 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.977+0000 7f718f3ba640 10 monclient(hunting): picked mon.noname-b con 0x7f71880b45a0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718f3ba640 10 monclient(hunting): picked mon.noname-a con 0x7f7188146450 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718f3ba640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7188140520 con 0x7f7188146450 2024-03-25T19:41:02.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718f3ba640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71881401f0 con 0x7f71880b45a0 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718f3ba640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7188135cb0 con 0x7f71880a9920 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718f3ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718f3ba640 10 monclient(hunting): authenticate will time out at 4838.242318s 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718d156640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f71880a9920 0x7f7188140da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:40844/0 (socket says 172.21.15.50:40844) 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.978+0000 7f718d156640 1 -- 172.21.15.50:0/764255733 learned_addr learned my addr 172.21.15.50:0/764255733 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.979+0000 7f717dffb640 10 client.?.objecter ms_handle_connect 0x7f7188146450 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.979+0000 7f717dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.979+0000 7f717dffb640 10 client.?.objecter ms_handle_connect 0x7f71880a9920 2024-03-25T19:41:02.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.979+0000 7f717dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342443071 0 0) 0x7f7188140520 con 0x7f7188146450 2024-03-25T19:41:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): my global_id is 4167 2024-03-25T19:41:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f716c004eb0 con 0x7f7188146450 2024-03-25T19:41:02.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872060980 0 0) 0x7f7188135cb0 con 0x7f71880a9920 2024-03-25T19:41:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 10 monclient(hunting): my global_id is 4166 2024-03-25T19:41:02.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.983+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7188140520 con 0x7f71880a9920 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469525400 0 0) 0x7f7188140520 con 0x7f71880a9920 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7188135cb0 con 0x7f71880a9920 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f71840018b0 con 0x7f71880a9920 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 10 monclient(hunting): dump: 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542666605 0 0) 0x7f716c004eb0 con 0x7f7188146450 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7188140520 con 0x7f7188146450 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7180001b10 con 0x7f7188146450 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:02.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.990+0000 7f717dffb640 10 monclient(hunting): dump: 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:02.992 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:41:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.994+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3642806375 0 0) 0x7f7188135cb0 con 0x7f71880a9920 2024-03-25T19:41:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.994+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 >> v1:172.21.15.76:6789/0 conn(0x7f71880b45a0 legacy=0x7f7188134e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.994+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 >> v1:172.21.15.50:6789/0 conn(0x7f7188146450 legacy=0x7f71881355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.994+0000 7f717dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f717dffb640 1 monclient: found mon.c 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f717dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f717dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71881425c0 con 0x7f71880a9920 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f717dffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f717dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f717dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 5 monclient: authenticate success, global_id 4166 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 10 monclient: _renew_subs 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71881428a0 con 0x7f71880a9920 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:02.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 10 monclient: _renew_subs 2024-03-25T19:41:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7188140b90 con 0x7f71880a9920 2024-03-25T19:41:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.995+0000 7f718f3ba640 1 librados: init done 2024-03-25T19:41:02.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.996+0000 7f718f3ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.996+0000 7f718f3ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.997+0000 7f718f3ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.997+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71880b7db0 con 0x7f71880a9920 2024-03-25T19:41:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.998+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71840047e0 con 0x7f71880a9920 2024-03-25T19:41:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.998+0000 7f717dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:02.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:02.998+0000 7f717f7fe640 4 set_mon_vals no callback set 2024-03-25T19:41:03.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.137+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7184004b30 con 0x7f71880a9920 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.137+0000 7f717dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.137+0000 7f717dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.137+0000 7f717dffb640 10 monclient: dump: 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:epoch 1 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:03.137 INFO:teuthology.orchestra.run.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:03.138 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:41:03.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.138+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 4101010985 0 0) 0x7f7184004d10 con 0x7f71880a9920 2024-03-25T19:41:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.191+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f7184014b10 con 0x7f71880a9920 2024-03-25T19:41:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.191+0000 7f717dffb640 10 client.4166.objecter ms_dispatch 0x7f7188000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.191+0000 7f717dffb640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:03.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.191+0000 7f717dffb640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.191+0000 7f717dffb640 20 client.4166.objecter dump_active .. 0 homeless 2024-03-25T19:41:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.191+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7184013110 con 0x7f71880a9920 2024-03-25T19:41:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.192+0000 7f717dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.192+0000 7f717dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:03.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.315+0000 7f718f3ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.316+0000 7f718f3ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.316+0000 7f718f3ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:03.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.316+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f718813ea50 con 0x7f71880a9920 2024-03-25T19:41:03.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.627+0000 7f717dffb640 1 -- 172.21.15.50:0/764255733 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f7184013110 con 0x7f71880a9920 2024-03-25T19:41:03.627 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-03-25T19:41:03.627 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:03.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.627+0000 7f717dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:03.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.627+0000 7f717dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:03.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 10 librados: watch_flush enter 2024-03-25T19:41:03.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 10 librados: watch_flush exit 2024-03-25T19:41:03.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 20 client.4166.objecter shutdown clearing up homeless session... 2024-03-25T19:41:03.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 10 client.4166.objecter successfully canceled tick 2024-03-25T19:41:03.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 10 monclient: shutdown 2024-03-25T19:41:03.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:03.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 >> v1:172.21.15.50:6790/0 conn(0x7f71880a9920 legacy=0x7f7188140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:03.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 shutdown_connections 2024-03-25T19:41:03.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 >> 172.21.15.50:0/764255733 conn(0x7f71880b7f40 msgr2=0x7f7188074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:03.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 shutdown_connections 2024-03-25T19:41:03.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 1 -- 172.21.15.50:0/764255733 wait complete. 2024-03-25T19:41:03.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-03-25T19:41:03.629+0000 7f718f3ba640 1 librados: shutdown 2024-03-25T19:41:03.644 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:41:03.644 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:41:04.241 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:04.242+0000 7fe5e5e10640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:41:04.260 INFO:tasks.ceph.mgr.x.smithi076.stderr:2024-03-25T19:41:04.261+0000 7fe5e5e10640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:41:04.449 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.601+0000 7f642d73b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.601+0000 7f642d73b640 10 monclient: build_initial_monmap 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.601+0000 7f642d73b640 10 monclient: monmap: 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:04.602689+0000 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:04.602689+0000 2024-03-25T19:41:04.601 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:04.602 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:04.602 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:04.602 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:04.602 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:04.602 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.602+0000 7f642d73b640 1 Processor -- start 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.602+0000 7f642d73b640 1 -- start start 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.602+0000 7f642d73b640 10 monclient: init 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 10 monclient(hunting): picked mon.noname-a con 0x7f64280a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 10 monclient(hunting): picked mon.noname-b con 0x7f642813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 10 monclient(hunting): picked mon.noname-c con 0x7f64281430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6428142520 con 0x7f64280a98f0 2024-03-25T19:41:04.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64280b28a0 con 0x7f64281430c0 2024-03-25T19:41:04.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64280b2a30 con 0x7f642813f710 2024-03-25T19:41:04.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:04.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.603+0000 7f642d73b640 10 monclient(hunting): authenticate will time out at 4839.867512s 2024-03-25T19:41:04.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.605+0000 7f64267fc640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f642813f710 0x7f64281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:40858/0 (socket says 172.21.15.50:40858) 2024-03-25T19:41:04.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.605+0000 7f64267fc640 1 -- 172.21.15.50:0/718975219 learned_addr learned my addr 172.21.15.50:0/718975219 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:04.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.617+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942003157 0 0) 0x7f6428142520 con 0x7f64280a98f0 2024-03-25T19:41:04.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.617+0000 7f6425ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.617+0000 7f6425ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.617+0000 7f6425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.617+0000 7f6425ffb640 10 monclient(hunting): my global_id is 4185 2024-03-25T19:41:04.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.617+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6414003520 con 0x7f64280a98f0 2024-03-25T19:41:04.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.698+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1776839434 0 0) 0x7f6414003520 con 0x7f64280a98f0 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.698+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 >> v1:172.21.15.50:6790/0 conn(0x7f642813f710 legacy=0x7f64281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.698+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 >> v1:172.21.15.76:6789/0 conn(0x7f64281430c0 legacy=0x7f642814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 1 monclient: found mon.noname-a 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64280b44c0 con 0x7f64280a98f0 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6418001690 con 0x7f64280a98f0 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f6425ffb640 10 monclient: dump: 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f642d73b640 5 monclient: authenticate success, global_id 4185 2024-03-25T19:41:04.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.699+0000 7f642d73b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:04.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.702+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64180018b0 con 0x7f64280a98f0 2024-03-25T19:41:04.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.702+0000 7f6425ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:04.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.702+0000 7f642d73b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:04.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.702+0000 7f642d73b640 4 set_mon_vals no callback set 2024-03-25T19:41:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f6425ffb640 1 -- 172.21.15.50:0/718975219 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6418001c40 con 0x7f64280a98f0 2024-03-25T19:41:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f6425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f6425ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f6425ffb640 10 monclient: dump: 2024-03-25T19:41:04.703 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f642d73b640 10 monclient: shutdown 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f642d73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f642d73b640 1 -- 172.21.15.50:0/718975219 >> v1:172.21.15.50:6789/0 conn(0x7f64280a98f0 legacy=0x7f64280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f642d73b640 1 -- 172.21.15.50:0/718975219 shutdown_connections 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.703+0000 7f642d73b640 1 -- 172.21.15.50:0/718975219 >> 172.21.15.50:0/718975219 conn(0x7f64280b7ea0 msgr2=0x7f64280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.704+0000 7f642d73b640 1 -- 172.21.15.50:0/718975219 shutdown_connections 2024-03-25T19:41:04.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.704+0000 7f642d73b640 1 -- 172.21.15.50:0/718975219 wait complete. 2024-03-25T19:41:04.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.705+0000 7f642d73b640 10 monclient: build_initial_monmap 2024-03-25T19:41:04.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.705+0000 7f642d73b640 10 monclient: monmap: 2024-03-25T19:41:04.705 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:04.705 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.706+0000 7f642d73b640 1 librados: starting msgr at 2024-03-25T19:41:04.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.706+0000 7f642d73b640 1 librados: starting objecter 2024-03-25T19:41:04.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.706+0000 7f642d73b640 1 Processor -- start 2024-03-25T19:41:04.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.706+0000 7f642d73b640 1 -- start start 2024-03-25T19:41:04.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.706+0000 7f642d73b640 1 librados: setting wanted keys 2024-03-25T19:41:04.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.706+0000 7f642d73b640 1 librados: calling monclient init 2024-03-25T19:41:04.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.706+0000 7f642d73b640 10 monclient: init 2024-03-25T19:41:04.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:04.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:04.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 10 monclient(hunting): picked mon.noname-a con 0x7f64280a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 10 monclient(hunting): picked mon.noname-c con 0x7f642813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 10 monclient(hunting): picked mon.noname-b con 0x7f64281430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642813b9e0 con 0x7f64280a98f0 2024-03-25T19:41:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642814dd30 con 0x7f64281430c0 2024-03-25T19:41:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6428135680 con 0x7f642813f710 2024-03-25T19:41:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.707+0000 7f642d73b640 10 monclient(hunting): authenticate will time out at 4839.971681s 2024-03-25T19:41:04.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.708+0000 7f6426ffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f64280a98f0 0x7f642814e3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39482/0 (socket says 172.21.15.50:39482) 2024-03-25T19:41:04.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.708+0000 7f6426ffd640 1 -- 172.21.15.50:0/3154884652 learned_addr learned my addr 172.21.15.50:0/3154884652 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:04.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.709+0000 7f641f7fe640 10 client.?.objecter ms_handle_connect 0x7f64280a98f0 2024-03-25T19:41:04.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.709+0000 7f641f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:04.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.709+0000 7f641f7fe640 10 client.?.objecter ms_handle_connect 0x7f64281430c0 2024-03-25T19:41:04.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.709+0000 7f641f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.712+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291835366 0 0) 0x7f642814dd30 con 0x7f64281430c0 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.712+0000 7f641f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 10 monclient(hunting): my global_id is 4171 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6408004eb0 con 0x7f64281430c0 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831201634 0 0) 0x7f642813b9e0 con 0x7f64280a98f0 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 10 monclient(hunting): my global_id is 4188 2024-03-25T19:41:04.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.713+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f642814dd30 con 0x7f64280a98f0 2024-03-25T19:41:04.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.720+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 301845485 0 0) 0x7f6408004eb0 con 0x7f64281430c0 2024-03-25T19:41:04.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.721+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f642813b9e0 con 0x7f64281430c0 2024-03-25T19:41:04.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.721+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6420001b90 con 0x7f64281430c0 2024-03-25T19:41:04.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.721+0000 7f641f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.721+0000 7f641f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.721+0000 7f641f7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:04.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.721+0000 7f641f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.723+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1435308430 0 0) 0x7f642814dd30 con 0x7f64280a98f0 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.723+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6408004eb0 con 0x7f64280a98f0 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.723+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6418001690 con 0x7f64280a98f0 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.723+0000 7f641f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.723+0000 7f641f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.723+0000 7f641f7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:04.723 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:04.724 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:04.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.725+0000 7f641f7fe640 10 client.?.objecter ms_handle_connect 0x7f642813f710 2024-03-25T19:41:04.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:04.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2176517538 0 0) 0x7f642813b9e0 con 0x7f64281430c0 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 >> v1:172.21.15.50:6790/0 conn(0x7f642813f710 legacy=0x7f6428134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 >> v1:172.21.15.50:6789/0 conn(0x7f64280a98f0 legacy=0x7f642814e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 1 monclient: found mon.b 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642813be70 con 0x7f64281430c0 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f641f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 5 monclient: authenticate success, global_id 4171 2024-03-25T19:41:04.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 10 monclient: _renew_subs 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642814caa0 con 0x7f64281430c0 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 10 monclient: _renew_subs 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f642814c730 con 0x7f64281430c0 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.726+0000 7f642d73b640 1 librados: init done 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.728+0000 7f642d73b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.728+0000 7f642d73b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.728+0000 7f642d73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:04.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.728+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642814ce40 con 0x7f64281430c0 2024-03-25T19:41:04.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.729+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6420001cf0 con 0x7f64281430c0 2024-03-25T19:41:04.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.729+0000 7f641f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:04.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.729+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6420003b10 con 0x7f64281430c0 2024-03-25T19:41:04.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.729+0000 7f641f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:04.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.729+0000 7f641f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:04.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.729+0000 7f641f7fe640 10 monclient: dump: 2024-03-25T19:41:04.731 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:04.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:04.729+0000 7f64257fa640 4 set_mon_vals no callback set 2024-03-25T19:41:05.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.036+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 860193352 0 0) 0x7f6420003c70 con 0x7f64281430c0 2024-03-25T19:41:05.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.037+0000 7f6426ffd640 10 monclient: get_auth_request con 0x7f640806dd60 auth_method 0 2024-03-25T19:41:05.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.037+0000 7f641f7fe640 10 client.4171.objecter ms_handle_connect 0x7f640806dd60 2024-03-25T19:41:05.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.168+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f6420053810 con 0x7f64281430c0 2024-03-25T19:41:05.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.168+0000 7f641f7fe640 10 client.4171.objecter ms_dispatch 0x7f6428000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:05.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.168+0000 7f641f7fe640 3 client.4171.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:05.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.168+0000 7f641f7fe640 3 client.4171.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:05.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.168+0000 7f641f7fe640 20 client.4171.objecter dump_active .. 0 homeless 2024-03-25T19:41:05.174 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.175+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f642004f320 con 0x7f64281430c0 2024-03-25T19:41:05.174 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.175+0000 7f641f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:05.174 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.175+0000 7f641f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:05.537 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.538+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f642004edd0 con 0x7f64281430c0 2024-03-25T19:41:05.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.552+0000 7f642d73b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.552+0000 7f642d73b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.552+0000 7f642d73b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:05.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.552+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64280627b0 con 0x7f64281430c0 2024-03-25T19:41:05.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.656+0000 7f641f7fe640 1 -- 172.21.15.50:0/3154884652 <== mon.1 v1:172.21.15.76:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f6420003970 con 0x7f64281430c0 2024-03-25T19:41:05.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.656+0000 7f641f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:05.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.656+0000 7f641f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:05.656 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:05.656 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:05.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 10 librados: watch_flush enter 2024-03-25T19:41:05.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 10 librados: watch_flush exit 2024-03-25T19:41:05.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 20 client.4171.objecter shutdown clearing up homeless session... 2024-03-25T19:41:05.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 10 client.4171.objecter successfully canceled tick 2024-03-25T19:41:05.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 >> v1:172.21.15.76:6800/2572556811 conn(0x7f640806dd60 legacy=0x7f6408070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 10 monclient: shutdown 2024-03-25T19:41:05.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:05.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 >> v1:172.21.15.76:6789/0 conn(0x7f64281430c0 legacy=0x7f6428134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:05.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 shutdown_connections 2024-03-25T19:41:05.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 >> 172.21.15.50:0/3154884652 conn(0x7f64280b7ea0 msgr2=0x7f6428074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:05.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 shutdown_connections 2024-03-25T19:41:05.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 1 -- 172.21.15.50:0/3154884652 wait complete. 2024-03-25T19:41:05.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:05.659+0000 7f642d73b640 1 librados: shutdown 2024-03-25T19:41:05.673 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:09.310 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:06.853 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.371 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:06.915 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.399 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:06.936 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.455 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:06.996 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.460 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:06.997 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.515 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:07.056 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.526 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:07.070 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.585 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:07.129 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.799 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:07.343 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.817 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:07.361 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.862 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:07.405 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.871 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:07.397 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.879 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:07.423 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.892 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:07.418 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.921 INFO:tasks.ceph.osd.5.smithi076.stderr:2024-03-25T19:41:09.913+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:09.932 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:07.458 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:09.953 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:07.478 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.002 INFO:tasks.ceph.osd.7.smithi076.stderr:2024-03-25T19:41:09.996+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:10.066 INFO:tasks.ceph.osd.6.smithi076.stderr:2024-03-25T19:41:10.057+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:10.140 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-03-25T19:41:10.134+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:10.371 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:07.915 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.410 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-03-25T19:41:10.405+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:10.428 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-03-25T19:41:10.423+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:10.468 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:08.006 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.481 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-03-25T19:41:10.476+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:10.507 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-03-25T19:41:10.502+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:10.521 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:08.062 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.611 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:08.155 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.733 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:08.276 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.837 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:08.375 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.886 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:08.427 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.943 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:08.469 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.958 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:08.501 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.971 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:08.497 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.976 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:08.520 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:10.977 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:08.521 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.085 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:08.629 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.150 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:08.687 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.238 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:08.779 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.317 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:08.842 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.327 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:08.872 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.332 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:08.875 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.355 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:08.881 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.365 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:08.909 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.372 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:08.915 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.416 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:08.953 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.527 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:09.068 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.615 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:09.159 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.632 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:09.157 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.655 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:09.198 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.663 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:09.206 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.694 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:09.220 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.697 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:09.235 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.751 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:09.294 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.822 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:09.363 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.898 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:09.423 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.906 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:09.450 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.921 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:09.465 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.956 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:09.499 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.973 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:09.511 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:11.983 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:09.509 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.043 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:09.587 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.114 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:09.655 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.348 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:09.687 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.349 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:09.741 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.349 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:09.736 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.349 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:09.779 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.350 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:09.798 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.350 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:09.794 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.350 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:09.879 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.401 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:09.942 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.430 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:09.955 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.462 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:10.005 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.491 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:10.035 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.517 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:10.054 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.544 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:10.087 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.561 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.087 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.624 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:10.168 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.649 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:12.692 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:10.233 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.711 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:10.237 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.738+0000 7fc66e884640 10 monclient: get_monmap_and_config 2024-03-25T19:41:12.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.738+0000 7fc66e884640 10 monclient: build_initial_monmap 2024-03-25T19:41:12.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.738+0000 7fc66e884640 10 monclient: monmap: 2024-03-25T19:41:12.737 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:12.739207+0000 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:12.739207+0000 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:12.738 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:12.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 1 Processor -- start 2024-03-25T19:41:12.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 1 -- start start 2024-03-25T19:41:12.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient: init 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient(hunting): picked mon.noname-b con 0x7fc6680b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient(hunting): picked mon.noname-c con 0x7fc6680b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient(hunting): picked mon.noname-a con 0x7fc66814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6680af940 con 0x7fc66814be90 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6680b0a70 con 0x7fc6680b1730 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6680b0d50 con 0x7fc6680b4d60 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.739+0000 7fc66e884640 10 monclient(hunting): authenticate will time out at 4848.003473s 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.740+0000 7fc667fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fc6680b4d60 0x7fc66813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:46720/0 (socket says 172.21.15.50:46720) 2024-03-25T19:41:12.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.740+0000 7fc667fff640 1 -- 172.21.15.50:0/3539605186 learned_addr learned my addr 172.21.15.50:0/3539605186 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218852212 0 0) 0x7fc6680b0d50 con 0x7fc6680b4d60 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): my global_id is 4199 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc650003520 con 0x7fc6680b4d60 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506574756 0 0) 0x7fc6680b0a70 con 0x7fc6680b1730 2024-03-25T19:41:12.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): my global_id is 4198 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6680b0d50 con 0x7fc6680b1730 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133164262 0 0) 0x7fc6680af940 con 0x7fc66814be90 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:41:12.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.742+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6680b0a70 con 0x7fc66814be90 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820542321 0 0) 0x7fc6680b0d50 con 0x7fc6680b1730 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 >> v1:172.21.15.50:6790/0 conn(0x7fc6680b4d60 legacy=0x7fc66813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 >> v1:172.21.15.50:6789/0 conn(0x7fc66814be90 legacy=0x7fc66814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 1 monclient: found mon.noname-c 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6680b1030 con 0x7fc6680b1730 2024-03-25T19:41:12.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:12.746 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:10.288 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc66e884640 5 monclient: authenticate success, global_id 4198 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc65c0015f0 con 0x7fc6680b1730 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc66e884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc666ffd640 10 monclient: dump: 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.748 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.745+0000 7fc66e884640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc65c001750 con 0x7fc6680b1730 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc666ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc66e884640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc66e884640 4 set_mon_vals no callback set 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc666ffd640 1 -- 172.21.15.50:0/3539605186 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc65c001b60 con 0x7fc6680b1730 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc666ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc666ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc666ffd640 10 monclient: dump: 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:12.749 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc66e884640 10 monclient: shutdown 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc66e884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc66e884640 1 -- 172.21.15.50:0/3539605186 >> v1:172.21.15.76:6789/0 conn(0x7fc6680b1730 legacy=0x7fc6680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc66e884640 1 -- 172.21.15.50:0/3539605186 shutdown_connections 2024-03-25T19:41:12.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.747+0000 7fc66e884640 1 -- 172.21.15.50:0/3539605186 >> 172.21.15.50:0/3539605186 conn(0x7fc6680617a0 msgr2=0x7fc668061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.748+0000 7fc66e884640 1 -- 172.21.15.50:0/3539605186 shutdown_connections 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.748+0000 7fc66e884640 1 -- 172.21.15.50:0/3539605186 wait complete. 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.748+0000 7fc66e884640 10 monclient: build_initial_monmap 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.748+0000 7fc66e884640 10 monclient: monmap: 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:12.751 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:12.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.748+0000 7fc66e884640 1 librados: starting msgr at 2024-03-25T19:41:12.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.748+0000 7fc66e884640 1 librados: starting objecter 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 1 Processor -- start 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 1 -- start start 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 1 librados: setting wanted keys 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 1 librados: calling monclient init 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient: init 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient(hunting): picked mon.noname-a con 0x7fc6680b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:12.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient(hunting): picked mon.noname-c con 0x7fc6680b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient(hunting): picked mon.noname-b con 0x7fc66814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc668139970 con 0x7fc6680b1730 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc668138750 con 0x7fc66814be90 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc668139bc0 con 0x7fc6680b4d60 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.749+0000 7fc66e884640 10 monclient(hunting): authenticate will time out at 4848.013589s 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc6677fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fc6680b4d60 0x7fc668144240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:46724/0 (socket says 172.21.15.50:46724) 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc667fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fc6680b1730 0x7fc668137f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48622/0 (socket says 172.21.15.50:48622) 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc6677fe640 1 -- 172.21.15.50:0/3696900104 learned_addr learned my addr 172.21.15.50:0/3696900104 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc664ff9640 10 client.?.objecter ms_handle_connect 0x7fc6680b4d60 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc664ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc664ff9640 10 client.?.objecter ms_handle_connect 0x7fc6680b1730 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc664ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:12.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc664ff9640 10 client.?.objecter ms_handle_connect 0x7fc66814be90 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.750+0000 7fc664ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.752+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566122690 0 0) 0x7fc668138750 con 0x7fc66814be90 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.752+0000 7fc664ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.752+0000 7fc664ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.752+0000 7fc664ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.752+0000 7fc664ff9640 10 monclient(hunting): my global_id is 4201 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.752+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc654004eb0 con 0x7fc66814be90 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.752+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458348162 0 0) 0x7fc668139970 con 0x7fc6680b1730 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc668138750 con 0x7fc6680b1730 2024-03-25T19:41:12.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205707802 0 0) 0x7fc668139bc0 con 0x7fc6680b4d60 2024-03-25T19:41:12.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:12.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:12.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:12.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:41:12.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.753+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc668139970 con 0x7fc6680b4d60 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2401596391 0 0) 0x7fc654004eb0 con 0x7fc66814be90 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc668139bc0 con 0x7fc66814be90 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc660001bd0 con 0x7fc66814be90 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.758 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345917575 0 0) 0x7fc668138750 con 0x7fc6680b1730 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc654004eb0 con 0x7fc6680b1730 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc6580018d0 con 0x7fc6680b1730 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.757+0000 7fc664ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:12.759 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.758+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 792309275 0 0) 0x7fc668139970 con 0x7fc6680b4d60 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.758+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc668138750 con 0x7fc6680b4d60 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.758+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc65c001a60 con 0x7fc6680b4d60 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.758+0000 7fc664ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:12.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.758+0000 7fc664ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.758+0000 7fc664ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:12.761 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232579602 0 0) 0x7fc654004eb0 con 0x7fc6680b1730 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 >> v1:172.21.15.50:6790/0 conn(0x7fc6680b4d60 legacy=0x7fc668144240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 >> v1:172.21.15.76:6789/0 conn(0x7fc66814be90 legacy=0x7fc668144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 1 monclient: found mon.a 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc66813aff0 con 0x7fc6680b1730 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc664ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:12.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 5 monclient: authenticate success, global_id 4218 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 10 monclient: _renew_subs 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc668137850 con 0x7fc6680b1730 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 10 monclient: _renew_subs 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6681e3670 con 0x7fc6680b1730 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.760+0000 7fc66e884640 1 librados: init done 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.761+0000 7fc66e884640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.761+0000 7fc66e884640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.761+0000 7fc66e884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.761+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6680627b0 con 0x7fc6680b1730 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.763+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc658001d00 con 0x7fc6680b1730 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.763+0000 7fc664ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:12.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:12.763+0000 7fc6667fc640 4 set_mon_vals no callback set 2024-03-25T19:41:12.782 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:10.326 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.786 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:10.324 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.838 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:10.381 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.850 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.375 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.921 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:10.464 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.976 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:10.501 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:12.979 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:10.520 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.013 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:10.555 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.038+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc658004a00 con 0x7fc6680b1730 2024-03-25T19:41:13.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.038+0000 7fc664ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:13.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.038+0000 7fc664ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:13.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.038+0000 7fc664ff9640 10 monclient: dump: 2024-03-25T19:41:13.039 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:13.039 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:13.039 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:13.039 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:13.039 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:13.039 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:13.040 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:13.040 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:13.040 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:13.040 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:13.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.038+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fc65801aa50 con 0x7fc6680b1730 2024-03-25T19:41:13.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.039+0000 7fc6677fe640 10 monclient: get_auth_request con 0x7fc65406e0c0 auth_method 0 2024-03-25T19:41:13.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.040+0000 7fc664ff9640 10 client.4218.objecter ms_handle_connect 0x7fc65406e0c0 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fc658054830 con 0x7fc6680b1730 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 10 client.4218.objecter ms_dispatch 0x7fc668000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc658050170 con 0x7fc6680b1730 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:13.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.043+0000 7fc664ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:13.088 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:10.632 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.147 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.672 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.228+0000 7fc66e884640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.228+0000 7fc66e884640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.228+0000 7fc66e884640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:13.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.228+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc66813b180 con 0x7fc6680b1730 2024-03-25T19:41:13.249 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:10.792 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.257 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:10.783 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.275+0000 7fc664ff9640 1 -- 172.21.15.50:0/3696900104 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fc658052270 con 0x7fc6680b1730 2024-03-25T19:41:13.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.275+0000 7fc664ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:13.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.275+0000 7fc664ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:13.275 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:13.275 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:13.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 10 librados: watch_flush enter 2024-03-25T19:41:13.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 10 librados: watch_flush exit 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 10 client.4218.objecter successfully canceled tick 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 >> v1:172.21.15.76:6800/2572556811 conn(0x7fc65406e0c0 legacy=0x7fc654070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 10 monclient: shutdown 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.277+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 >> v1:172.21.15.50:6789/0 conn(0x7fc6680b1730 legacy=0x7fc668137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.278+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 shutdown_connections 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.278+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 >> 172.21.15.50:0/3696900104 conn(0x7fc6680617a0 msgr2=0x7fc66813c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.278+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 shutdown_connections 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.278+0000 7fc66e884640 1 -- 172.21.15.50:0/3696900104 wait complete. 2024-03-25T19:41:13.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:13.278+0000 7fc66e884640 1 librados: shutdown 2024-03-25T19:41:13.281 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:10.825 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.283 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:10.820 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.287 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:13.341 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:10.878 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.400 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:10.944 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.436 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:10.962 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.493 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:11.036 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.541 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:11.082 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.546 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:11.089 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.595 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:11.136 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.634 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:11.178 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.688 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:11.232 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.739 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:11.264 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.775 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:11.319 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.795 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:11.320 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:13.831 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:11.375 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:14.024 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:11.550 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:14.084 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:11.609 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:14.123 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:11.667 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:14.179 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:11.723 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.230 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:16.773 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.279 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:16.820 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.292 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:16.836 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.462 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:16.988 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.507 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:17.033 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.543 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:17.080 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.549 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:17.093 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:19.595 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:17.139 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:20.093 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.222+0000 7f40619d6640 10 monclient: get_monmap_and_config 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.222+0000 7f40619d6640 10 monclient: build_initial_monmap 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.222+0000 7f40619d6640 10 monclient: monmap: 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:20.223428+0000 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:20.223428+0000 2024-03-25T19:41:20.222 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:20.223 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:20.223 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:20.223 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:20.223 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:20.223 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.224 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.223+0000 7f40619d6640 1 Processor -- start 2024-03-25T19:41:20.224 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.223+0000 7f40619d6640 1 -- start start 2024-03-25T19:41:20.224 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.223+0000 7f40619d6640 10 monclient: init 2024-03-25T19:41:20.224 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:20.224 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 10 monclient(hunting): picked mon.noname-b con 0x7f405c0b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 10 monclient(hunting): picked mon.noname-a con 0x7f405c0b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 10 monclient(hunting): picked mon.noname-c con 0x7f405c14be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0af940 con 0x7f405c0b1730 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0b0a70 con 0x7f405c14be90 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0b0d50 con 0x7f405c0b4d60 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.224+0000 7f40619d6640 10 monclient(hunting): authenticate will time out at 4855.488126s 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.225+0000 7f405b7fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f405c14be90 0x7f405c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:33064/0 (socket says 172.21.15.50:33064) 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.225+0000 7f405b7fe640 1 -- 172.21.15.50:0/3331455979 learned_addr learned my addr 172.21.15.50:0/3331455979 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:20.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.225+0000 7f405a7fc640 1 --1- 172.21.15.50:0/3331455979 >> v1:172.21.15.50:6789/0 conn(0x7f405c0b1730 0x7f405c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33270/0 (socket says 172.21.15.50:33270) 2024-03-25T19:41:20.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809962043 0 0) 0x7f405c0af940 con 0x7f405c0b1730 2024-03-25T19:41:20.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4044003520 con 0x7f405c0b1730 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964107277 0 0) 0x7f405c0b0a70 con 0x7f405c14be90 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.227+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c0af940 con 0x7f405c14be90 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.228+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019139510 0 0) 0x7f405c0b0d50 con 0x7f405c0b4d60 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.228+0000 7f4059ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.228+0000 7f4059ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.228+0000 7f4059ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.228+0000 7f4059ffb640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:41:20.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.228+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c0b0a70 con 0x7f405c0b4d60 2024-03-25T19:41:20.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3366752298 0 0) 0x7f405c0af940 con 0x7f405c14be90 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 >> v1:172.21.15.50:6790/0 conn(0x7f405c0b4d60 legacy=0x7f405c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 >> v1:172.21.15.50:6789/0 conn(0x7f405c0b1730 legacy=0x7f405c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 1 monclient: found mon.noname-c 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c0b1030 con 0x7f405c14be90 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f40619d6640 5 monclient: authenticate success, global_id 4204 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f40619d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:20.231 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f40540019b0 con 0x7f405c14be90 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f4059ffb640 10 monclient: dump: 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:20.232 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.233 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.231+0000 7f40619d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:20.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.233+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4054001c00 con 0x7f405c14be90 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.233+0000 7f4059ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.233+0000 7f40619d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.233+0000 7f40619d6640 4 set_mon_vals no callback set 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f4059ffb640 1 -- 172.21.15.50:0/3331455979 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4054001f90 con 0x7f405c14be90 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f4059ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f4059ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f4059ffb640 10 monclient: dump: 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.235 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f40619d6640 10 monclient: shutdown 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f40619d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f40619d6640 1 -- 172.21.15.50:0/3331455979 >> v1:172.21.15.76:6789/0 conn(0x7f405c14be90 legacy=0x7f405c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f40619d6640 1 -- 172.21.15.50:0/3331455979 shutdown_connections 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f40619d6640 1 -- 172.21.15.50:0/3331455979 >> 172.21.15.50:0/3331455979 conn(0x7f405c0617a0 msgr2=0x7f405c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f40619d6640 1 -- 172.21.15.50:0/3331455979 shutdown_connections 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.234+0000 7f40619d6640 1 -- 172.21.15.50:0/3331455979 wait complete. 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.235+0000 7f40619d6640 10 monclient: build_initial_monmap 2024-03-25T19:41:20.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.235+0000 7f40619d6640 10 monclient: monmap: 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:20.237 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.236+0000 7f40619d6640 1 librados: starting msgr at 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.236+0000 7f40619d6640 1 librados: starting objecter 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.236+0000 7f40619d6640 1 Processor -- start 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.236+0000 7f40619d6640 1 -- start start 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.236+0000 7f40619d6640 1 librados: setting wanted keys 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.236+0000 7f40619d6640 1 librados: calling monclient init 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.236+0000 7f40619d6640 10 monclient: init 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:20.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 10 monclient(hunting): picked mon.noname-b con 0x7f405c0b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 10 monclient(hunting): picked mon.noname-a con 0x7f405c0b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 10 monclient(hunting): picked mon.noname-c con 0x7f405c14be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c1d7960 con 0x7f405c0b4d60 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0aa860 con 0x7f405c0b1730 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c1d8500 con 0x7f405c14be90 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f405a7fc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f405c0b4d60 0x7f405c1d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33278/0 (socket says 172.21.15.50:33278) 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.238+0000 7f405a7fc640 1 -- 172.21.15.50:0/121944895 learned_addr learned my addr 172.21.15.50:0/121944895 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.237+0000 7f40619d6640 10 monclient(hunting): authenticate will time out at 4855.501661s 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.238+0000 7f4043fff640 10 client.?.objecter ms_handle_connect 0x7f405c0b4d60 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.238+0000 7f4043fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.238+0000 7f4043fff640 10 client.?.objecter ms_handle_connect 0x7f405c0b1730 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.238+0000 7f4043fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.238+0000 7f4043fff640 10 client.?.objecter ms_handle_connect 0x7f405c14be90 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.238+0000 7f4043fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.240+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240881610 0 0) 0x7f405c0aa860 con 0x7f405c0b1730 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.240+0000 7f4043fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.240+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.240+0000 7f4043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.240+0000 7f4043fff640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.240+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4048004eb0 con 0x7f405c0b1730 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622603405 0 0) 0x7f405c1d7960 con 0x7f405c0b4d60 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40480051d0 con 0x7f405c0b4d60 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384402604 0 0) 0x7f405c1d8500 con 0x7f405c14be90 2024-03-25T19:41:20.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:20.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:20.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:20.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:41:20.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.241+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c1d7960 con 0x7f405c14be90 2024-03-25T19:41:20.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.245+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1740235991 0 0) 0x7f4048004eb0 con 0x7f405c0b1730 2024-03-25T19:41:20.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.245+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f405c1d8500 con 0x7f405c0b1730 2024-03-25T19:41:20.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f404c001970 con 0x7f405c0b1730 2024-03-25T19:41:20.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 10 monclient(hunting): dump: 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2993931727 0 0) 0x7f405c1d7960 con 0x7f405c14be90 2024-03-25T19:41:20.247 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4048005cd0 con 0x7f405c14be90 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4054001f10 con 0x7f405c14be90 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.246+0000 7f4043fff640 10 monclient(hunting): dump: 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:20.248 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:20.249 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.248+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 869063255 0 0) 0x7f40480051d0 con 0x7f405c0b4d60 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.248+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f405c1d7960 con 0x7f405c0b4d60 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.248+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f40500018d0 con 0x7f405c0b4d60 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.248+0000 7f4043fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.248+0000 7f4043fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.248+0000 7f4043fff640 10 monclient(hunting): dump: 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:20.252 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3010501111 0 0) 0x7f4048005cd0 con 0x7f405c14be90 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 >> v1:172.21.15.76:6789/0 conn(0x7f405c0b1730 legacy=0x7f405c142b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 >> v1:172.21.15.50:6789/0 conn(0x7f405c0b4d60 legacy=0x7f405c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 1 monclient: found mon.c 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c1d8690 con 0x7f405c14be90 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:20.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f4043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 5 monclient: authenticate success, global_id 4208 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 10 monclient: _renew_subs 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c140380 con 0x7f405c14be90 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 10 monclient: _renew_subs 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c1386d0 con 0x7f405c14be90 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.250+0000 7f40619d6640 1 librados: init done 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.252+0000 7f40619d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.252+0000 7f40619d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.252+0000 7f40619d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:20.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.252+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c0627b0 con 0x7f405c14be90 2024-03-25T19:41:20.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40540015a0 con 0x7f405c14be90 2024-03-25T19:41:20.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f4043fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:20.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4054004e10 con 0x7f405c14be90 2024-03-25T19:41:20.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f4043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:20.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f4043fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:20.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f4043fff640 10 monclient: dump: 2024-03-25T19:41:20.316 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f405401ade0 con 0x7f405c14be90 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.315+0000 7f40597fa640 4 set_mon_vals no callback set 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.316+0000 7f405affd640 10 monclient: get_auth_request con 0x7f404806e5d0 auth_method 0 2024-03-25T19:41:20.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.316+0000 7f4043fff640 10 client.4208.objecter ms_handle_connect 0x7f404806e5d0 2024-03-25T19:41:20.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.519+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f4054054b20 con 0x7f405c14be90 2024-03-25T19:41:20.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.519+0000 7f4043fff640 10 client.4208.objecter ms_dispatch 0x7f405c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:20.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.519+0000 7f4043fff640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:20.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.519+0000 7f4043fff640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:20.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.519+0000 7f4043fff640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T19:41:20.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.524+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4054050460 con 0x7f405c14be90 2024-03-25T19:41:20.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.524+0000 7f4043fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:20.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.524+0000 7f4043fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:20.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.780+0000 7f40619d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:20.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.780+0000 7f40619d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:20.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.780+0000 7f40619d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:20.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.780+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f405c069f40 con 0x7f405c14be90 2024-03-25T19:41:20.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.804+0000 7f4043fff640 1 -- 172.21.15.50:0/121944895 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f4054052560 con 0x7f405c14be90 2024-03-25T19:41:20.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.804+0000 7f4043fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:20.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.804+0000 7f4043fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:20.804 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:20.805 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:20.807 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 10 librados: watch_flush enter 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 10 librados: watch_flush exit 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 10 client.4208.objecter successfully canceled tick 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 >> v1:172.21.15.76:6800/2572556811 conn(0x7f404806e5d0 legacy=0x7f4048070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 10 monclient: shutdown 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 >> v1:172.21.15.50:6790/0 conn(0x7f405c14be90 legacy=0x7f405c137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 shutdown_connections 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.807+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 >> 172.21.15.50:0/121944895 conn(0x7f405c0617a0 msgr2=0x7f405c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.808+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 shutdown_connections 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.808+0000 7f40619d6640 1 -- 172.21.15.50:0/121944895 wait complete. 2024-03-25T19:41:20.808 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:20.808+0000 7f40619d6640 1 librados: shutdown 2024-03-25T19:41:20.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:25.058 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:22.602 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.062 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:22.606 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.167 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:22.705 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.170 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:22.714 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.174 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:22.711 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.174 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:22.718 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.194 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:22.737 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.198 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:22.741 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.285 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:22.826 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.289 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:22.830 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.388 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:22.913 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.392 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:22.917 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.674 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:23.218 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.678 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:23.222 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.720 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:23.245 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:25.726 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:23.252 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:27.627 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:27.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.730+0000 7f0fa3f00640 10 monclient: get_monmap_and_config 2024-03-25T19:41:27.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.730+0000 7f0fa3f00640 10 monclient: build_initial_monmap 2024-03-25T19:41:27.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.730+0000 7f0fa3f00640 10 monclient: monmap: 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:27.731200+0000 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:27.731200+0000 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:27.730 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.730+0000 7f0fa3f00640 1 Processor -- start 2024-03-25T19:41:27.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 1 -- start start 2024-03-25T19:41:27.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 10 monclient: init 2024-03-25T19:41:27.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c0b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9c0b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c14be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0af940 con 0x7f0f9c14be90 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.731+0000 7f0fa3f00640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0b0a70 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.732+0000 7f0fa3f00640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0b0d50 con 0x7f0f9c0b1730 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.732+0000 7f0fa3f00640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.732+0000 7f0fa3f00640 10 monclient(hunting): authenticate will time out at 4862.995756s 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.732+0000 7f0fa1c9c640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0f9c0b4d60 0x7f0f9c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:57626/0 (socket says 172.21.15.50:57626) 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.732+0000 7f0fa149b640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f0f9c0b1730 0x7f0f9c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:41200/0 (socket says 172.21.15.50:41200) 2024-03-25T19:41:27.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.732+0000 7f0fa149b640 1 -- 172.21.15.50:0/3870135944 learned_addr learned my addr 172.21.15.50:0/3870135944 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723102671 0 0) 0x7f0f9c0b0a70 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f88003520 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444205860 0 0) 0x7f0f9c0af940 con 0x7f0f9c14be90 2024-03-25T19:41:27.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.735+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c0b0a70 con 0x7f0f9c14be90 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.736+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966574717 0 0) 0x7f0f9c0b0d50 con 0x7f0f9c0b1730 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.736+0000 7f0fa0c9a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.736+0000 7f0fa0c9a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.736+0000 7f0fa0c9a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.736+0000 7f0fa0c9a640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:41:27.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.736+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c0af940 con 0x7f0f9c0b1730 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 356698502 0 0) 0x7f0f9c0b0a70 con 0x7f0f9c14be90 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 >> v1:172.21.15.50:6790/0 conn(0x7f0f9c0b1730 legacy=0x7f0f9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 >> v1:172.21.15.76:6789/0 conn(0x7f0f9c0b4d60 legacy=0x7f0f9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 10 monclient: _finish_hunting 0 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 1 monclient: found mon.noname-a 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c0b1030 con 0x7f0f9c14be90 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 10 monclient: _finish_auth 0 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 10 monclient: _check_auth_tickets 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0f940019f0 con 0x7f0f9c14be90 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa0c9a640 10 monclient: dump: 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa3f00640 5 monclient: authenticate success, global_id 4227 2024-03-25T19:41:27.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.739+0000 7f0fa3f00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f94001b50 con 0x7f0f9c14be90 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa0c9a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa0c9a640 1 -- 172.21.15.50:0/3870135944 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0f94001f60 con 0x7f0f9c14be90 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa0c9a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa0c9a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa0c9a640 10 monclient: dump: 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:27.743 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa3f00640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.741+0000 7f0fa3f00640 4 set_mon_vals no callback set 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.742+0000 7f0fa3f00640 10 monclient: shutdown 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.742+0000 7f0fa3f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:27.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.742+0000 7f0fa3f00640 1 -- 172.21.15.50:0/3870135944 >> v1:172.21.15.50:6789/0 conn(0x7f0f9c14be90 legacy=0x7f0f9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.742+0000 7f0fa3f00640 1 -- 172.21.15.50:0/3870135944 shutdown_connections 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.742+0000 7f0fa3f00640 1 -- 172.21.15.50:0/3870135944 >> 172.21.15.50:0/3870135944 conn(0x7f0f9c0617a0 msgr2=0x7f0f9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.742+0000 7f0fa3f00640 1 -- 172.21.15.50:0/3870135944 shutdown_connections 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.742+0000 7f0fa3f00640 1 -- 172.21.15.50:0/3870135944 wait complete. 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.743+0000 7f0fa3f00640 10 monclient: build_initial_monmap 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.743+0000 7f0fa3f00640 10 monclient: monmap: 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:27.745 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.743+0000 7f0fa3f00640 1 librados: starting msgr at 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.743+0000 7f0fa3f00640 1 librados: starting objecter 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 1 Processor -- start 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 1 -- start start 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 1 librados: setting wanted keys 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 1 librados: calling monclient init 2024-03-25T19:41:27.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient: init 2024-03-25T19:41:27.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:27.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c0b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9c0b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c14be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0aabb0 con 0x7f0f9c0b1730 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0a9900 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0a9470 con 0x7f0f9c14be90 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.744+0000 7f0fa3f00640 10 monclient(hunting): authenticate will time out at 4863.008590s 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0fa149b640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0f9c0b4d60 0x7f0f9c136a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:57634/0 (socket says 172.21.15.50:57634) 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0fa149b640 1 -- 172.21.15.50:0/4042160759 learned_addr learned my addr 172.21.15.50:0/4042160759 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0f927fc640 10 client.?.objecter ms_handle_connect 0x7f0f9c0b4d60 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0f927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0f927fc640 10 client.?.objecter ms_handle_connect 0x7f0f9c14be90 2024-03-25T19:41:27.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0f927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0f927fc640 10 client.?.objecter ms_handle_connect 0x7f0f9c0b1730 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.745+0000 7f0f927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.747+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283116077 0 0) 0x7f0f9c0a9470 con 0x7f0f9c14be90 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.747+0000 7f0f927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.747+0000 7f0f927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.747+0000 7f0f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.747+0000 7f0f927fc640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.747+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80004eb0 con 0x7f0f9c14be90 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.748+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630165758 0 0) 0x7f0f9c0a9900 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.748+0000 7f0f927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.748+0000 7f0f927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.748+0000 7f0f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.748+0000 7f0f927fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:41:27.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.748+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c0a9470 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.749+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828074582 0 0) 0x7f0f9c0aabb0 con 0x7f0f9c0b1730 2024-03-25T19:41:27.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.749+0000 7f0f927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:27.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.749+0000 7f0f927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:27.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.749+0000 7f0f927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:27.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.749+0000 7f0f927fc640 10 monclient(hunting): my global_id is 4230 2024-03-25T19:41:27.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.749+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c0a9900 con 0x7f0f9c0b1730 2024-03-25T19:41:27.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.754+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948850393 0 0) 0x7f0f80004eb0 con 0x7f0f9c14be90 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.754+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f9c0aabb0 con 0x7f0f9c14be90 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.754+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0f94001e60 con 0x7f0f9c14be90 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.754+0000 7f0f927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.754+0000 7f0f927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 10 monclient(hunting): dump: 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:27.756 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871196110 0 0) 0x7f0f9c0a9470 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f80004eb0 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0f8c001870 con 0x7f0f9c0b4d60 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.755+0000 7f0f927fc640 10 monclient(hunting): dump: 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:27.757 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.756+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301294238 0 0) 0x7f0f9c0a9900 con 0x7f0f9c0b1730 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.756+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f9c0a9470 con 0x7f0f9c0b1730 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.756+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0f980017b0 con 0x7f0f9c0b1730 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.756+0000 7f0f927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.756+0000 7f0f927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.756+0000 7f0f927fc640 10 monclient(hunting): dump: 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:27.758 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:27.759 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:27.759 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:27.759 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2913056203 0 0) 0x7f0f9c0aabb0 con 0x7f0f9c14be90 2024-03-25T19:41:27.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 >> v1:172.21.15.76:6789/0 conn(0x7f0f9c0b4d60 legacy=0x7f0f9c136a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 >> v1:172.21.15.50:6789/0 conn(0x7f0f9c0b1730 legacy=0x7f0f9c140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:27.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 1 monclient: found mon.c 2024-03-25T19:41:27.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c0a9600 con 0x7f0f9c14be90 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0f927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 5 monclient: authenticate success, global_id 4214 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 10 monclient: _renew_subs 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c137830 con 0x7f0f9c14be90 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 10 monclient: _renew_subs 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:27.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c143610 con 0x7f0f9c14be90 2024-03-25T19:41:27.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.758+0000 7f0fa3f00640 1 librados: init done 2024-03-25T19:41:27.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.760+0000 7f0fa3f00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:27.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.760+0000 7f0fa3f00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:27.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.760+0000 7f0fa3f00640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:27.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.760+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c061c30 con 0x7f0f9c14be90 2024-03-25T19:41:27.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.784+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f94002020 con 0x7f0f9c14be90 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.784+0000 7f0f927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.784+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0f94003dc0 con 0x7f0f9c14be90 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.784+0000 7f0f927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.784+0000 7f0f927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.784+0000 7f0f927fc640 10 monclient: dump: 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:27.784 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:27.785 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:27.785 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:27.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.784+0000 7f0f93fff640 4 set_mon_vals no callback set 2024-03-25T19:41:27.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.785+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f0f94019d90 con 0x7f0f9c14be90 2024-03-25T19:41:27.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.786+0000 7f0fa1c9c640 10 monclient: get_auth_request con 0x7f0f8006e0b0 auth_method 0 2024-03-25T19:41:27.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.786+0000 7f0f927fc640 10 client.4214.objecter ms_handle_connect 0x7f0f8006e0b0 2024-03-25T19:41:27.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.788+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f0f94053840 con 0x7f0f9c14be90 2024-03-25T19:41:27.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.788+0000 7f0f927fc640 10 client.4214.objecter ms_dispatch 0x7f0f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:27.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.788+0000 7f0f927fc640 3 client.4214.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:27.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.788+0000 7f0f927fc640 3 client.4214.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:27.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:27.788+0000 7f0f927fc640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-25T19:41:28.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.215+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f9404f350 con 0x7f0f9c14be90 2024-03-25T19:41:28.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.215+0000 7f0f927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:28.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.215+0000 7f0f927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:28.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.440+0000 7f0fa3f00640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.440+0000 7f0fa3f00640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.440+0000 7f0fa3f00640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:41:28.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.440+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f9c0627b0 con 0x7f0f9c14be90 2024-03-25T19:41:28.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.456+0000 7f0f927fc640 1 -- 172.21.15.50:0/4042160759 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f0f94051280 con 0x7f0f9c14be90 2024-03-25T19:41:28.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.456+0000 7f0f927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:28.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.456+0000 7f0f927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:28.456 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:28.456 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 10 librados: watch_flush enter 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 10 librados: watch_flush exit 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 10 client.4214.objecter successfully canceled tick 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 >> v1:172.21.15.76:6800/2572556811 conn(0x7f0f8006e0b0 legacy=0x7f0f80070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 10 monclient: shutdown 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 >> v1:172.21.15.50:6790/0 conn(0x7f0f9c14be90 legacy=0x7f0f9c137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:28.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 shutdown_connections 2024-03-25T19:41:28.460 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 >> 172.21.15.50:0/4042160759 conn(0x7f0f9c0617a0 msgr2=0x7f0f9c13c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:28.460 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 shutdown_connections 2024-03-25T19:41:28.460 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.459+0000 7f0fa3f00640 1 -- 172.21.15.50:0/4042160759 wait complete. 2024-03-25T19:41:28.460 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:28.460+0000 7f0fa3f00640 1 librados: shutdown 2024-03-25T19:41:28.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:29.407 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:00:26.951 35340== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:29.452 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:00:26.995 35550== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:29.481 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:00:27.024 35341== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:29.510 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:00:27.051 35342== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:29.803 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:00:27.340 35343== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:29.841 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:00:27.366 35552== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:30.141 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:00:27.685 35549== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:30.154 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:00:27.679 35551== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:35.276 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.396+0000 7f825a486640 10 monclient: get_monmap_and_config 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.396+0000 7f825a486640 10 monclient: build_initial_monmap 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.396+0000 7f825a486640 10 monclient: monmap: 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:35.398420+0000 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:35.398420+0000 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:35.397 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:35.398 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:35.398 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:35.398 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:35.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.397+0000 7f825a486640 1 Processor -- start 2024-03-25T19:41:35.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.397+0000 7f825a486640 1 -- start start 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.397+0000 7f825a486640 10 monclient: init 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.397+0000 7f825a486640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 10 monclient(hunting): picked mon.noname-c con 0x7f82540b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 10 monclient(hunting): picked mon.noname-b con 0x7f82540b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 10 monclient(hunting): picked mon.noname-a con 0x7f825414be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82540af940 con 0x7f825414be90 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82540b0a70 con 0x7f82540b4d60 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82540b0d50 con 0x7f82540b1730 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f825a486640 10 monclient(hunting): authenticate will time out at 4870.662953s 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f8258a23640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f825414be90 0x7f825414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37222/0 (socket says 172.21.15.50:37222) 2024-03-25T19:41:35.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.398+0000 7f8258a23640 1 -- 172.21.15.50:0/894020553 learned_addr learned my addr 172.21.15.50:0/894020553 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139307721 0 0) 0x7f82540af940 con 0x7f825414be90 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): my global_id is 4233 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f823c003520 con 0x7f825414be90 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122467941 0 0) 0x7f82540b0d50 con 0x7f82540b1730 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82540af940 con 0x7f82540b1730 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172038028 0 0) 0x7f82540b0a70 con 0x7f82540b4d60 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:41:35.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.402+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82540b0d50 con 0x7f82540b4d60 2024-03-25T19:41:35.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 403813168 0 0) 0x7f823c003520 con 0x7f825414be90 2024-03-25T19:41:35.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 >> v1:172.21.15.50:6790/0 conn(0x7f82540b1730 legacy=0x7f82540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 >> v1:172.21.15.76:6789/0 conn(0x7f82540b4d60 legacy=0x7f825413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 1 monclient: found mon.noname-a 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82540b1030 con 0x7f825414be90 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f824c001a10 con 0x7f825414be90 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f8252ffd640 10 monclient: dump: 2024-03-25T19:41:35.407 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f825a486640 5 monclient: authenticate success, global_id 4233 2024-03-25T19:41:35.408 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.406+0000 7f825a486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:35.410 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f824c001b70 con 0x7f825414be90 2024-03-25T19:41:35.410 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f8252ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:35.410 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f8252ffd640 1 -- 172.21.15.50:0/894020553 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f824c001f80 con 0x7f825414be90 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f8252ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f8252ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f8252ffd640 10 monclient: dump: 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:35.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f825a486640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.408+0000 7f825a486640 4 set_mon_vals no callback set 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.409+0000 7f825a486640 10 monclient: shutdown 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.409+0000 7f825a486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.409+0000 7f825a486640 1 -- 172.21.15.50:0/894020553 >> v1:172.21.15.50:6789/0 conn(0x7f825414be90 legacy=0x7f825414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.409+0000 7f825a486640 1 -- 172.21.15.50:0/894020553 shutdown_connections 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.409+0000 7f825a486640 1 -- 172.21.15.50:0/894020553 >> 172.21.15.50:0/894020553 conn(0x7f82540617a0 msgr2=0x7f8254061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.409+0000 7f825a486640 1 -- 172.21.15.50:0/894020553 shutdown_connections 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.409+0000 7f825a486640 1 -- 172.21.15.50:0/894020553 wait complete. 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.410+0000 7f825a486640 10 monclient: build_initial_monmap 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.410+0000 7f825a486640 10 monclient: monmap: 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:35.412 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:35.413 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:35.413 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:35.413 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:35.413 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:35.413 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:35.413 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:35.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.411+0000 7f825a486640 1 librados: starting msgr at 2024-03-25T19:41:35.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.411+0000 7f825a486640 1 librados: starting objecter 2024-03-25T19:41:35.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.411+0000 7f825a486640 1 Processor -- start 2024-03-25T19:41:35.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.411+0000 7f825a486640 1 -- start start 2024-03-25T19:41:35.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.411+0000 7f825a486640 1 librados: setting wanted keys 2024-03-25T19:41:35.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.411+0000 7f825a486640 1 librados: calling monclient init 2024-03-25T19:41:35.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.411+0000 7f825a486640 10 monclient: init 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.412+0000 7f825a486640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.412+0000 7f825a486640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.412+0000 7f825a486640 10 monclient(hunting): picked mon.noname-a con 0x7f82540b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.412+0000 7f825a486640 10 monclient(hunting): picked mon.noname-c con 0x7f82540b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f825a486640 10 monclient(hunting): picked mon.noname-b con 0x7f825414be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f825a486640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8254145570 con 0x7f82540b1730 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f825a486640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8254144900 con 0x7f825414be90 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f825a486640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82541442e0 con 0x7f82540b4d60 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f8253fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f82540b1730 0x7f825413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37238/0 (socket says 172.21.15.50:37238) 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f8253fff640 1 -- 172.21.15.50:0/3049581978 learned_addr learned my addr 172.21.15.50:0/3049581978 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f825a486640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f825a486640 10 monclient(hunting): authenticate will time out at 4870.678246s 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f8250ff9640 10 client.?.objecter ms_handle_connect 0x7f82540b1730 2024-03-25T19:41:35.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.413+0000 7f8250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.414+0000 7f8250ff9640 10 client.?.objecter ms_handle_connect 0x7f82540b4d60 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.414+0000 7f8250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 10 client.?.objecter ms_handle_connect 0x7f825414be90 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783458990 0 0) 0x7f8254145570 con 0x7f82540b1730 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 10 monclient(hunting): my global_id is 4236 2024-03-25T19:41:35.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.415+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8240004eb0 con 0x7f82540b1730 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.416+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815581034 0 0) 0x7f82541442e0 con 0x7f82540b4d60 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.416+0000 7f8250ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.416+0000 7f8250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.416+0000 7f8250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.416+0000 7f8250ff9640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.416+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8254145570 con 0x7f82540b4d60 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.417+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202593182 0 0) 0x7f8254144900 con 0x7f825414be90 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.417+0000 7f8250ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.417+0000 7f8250ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.417+0000 7f8250ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.417+0000 7f8250ff9640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:41:35.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.417+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82541442e0 con 0x7f825414be90 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.421+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3253564327 0 0) 0x7f8240004eb0 con 0x7f82540b1730 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.421+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8254144900 con 0x7f82540b1730 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.421+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8244001a70 con 0x7f82540b1730 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.421+0000 7f8250ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.421+0000 7f8250ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.421+0000 7f8250ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:35.422 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.421+0000 7f8250ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:35.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.423+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 757137168 0 0) 0x7f8254145570 con 0x7f82540b4d60 2024-03-25T19:41:35.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.423+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8240004eb0 con 0x7f82540b4d60 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.425+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1273137974 0 0) 0x7f82541442e0 con 0x7f825414be90 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.425+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8240005e50 con 0x7f825414be90 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.425+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f824c001e80 con 0x7f825414be90 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.425+0000 7f8250ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.425+0000 7f8250ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.425+0000 7f8250ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.427 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 199799912 0 0) 0x7f8254144900 con 0x7f82540b1730 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 >> v1:172.21.15.50:6790/0 conn(0x7f82540b4d60 legacy=0x7f825412fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 >> v1:172.21.15.76:6789/0 conn(0x7f825414be90 legacy=0x7f82541e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 1 monclient: found mon.a 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82541457c0 con 0x7f82540b1730 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:35.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f8250ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 5 monclient: authenticate success, global_id 4236 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 10 monclient: _renew_subs 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8254144560 con 0x7f82540b1730 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 10 client.4236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 10 monclient: _renew_subs 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8254143c00 con 0x7f82540b1730 2024-03-25T19:41:35.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.426+0000 7f825a486640 1 librados: init done 2024-03-25T19:41:35.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.427+0000 7f825a486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:35.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.427+0000 7f825a486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:35.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.427+0000 7f825a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:35.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.427+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8254144750 con 0x7f82540b1730 2024-03-25T19:41:35.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.429+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8244001cd0 con 0x7f82540b1730 2024-03-25T19:41:35.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.429+0000 7f8250ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:35.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.429+0000 7f82527fc640 4 set_mon_vals no callback set 2024-03-25T19:41:35.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.773+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8244003a40 con 0x7f82540b1730 2024-03-25T19:41:35.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.773+0000 7f8250ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:35.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.773+0000 7f8250ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.773+0000 7f8250ff9640 10 monclient: dump: 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:35.774 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:35.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.866+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f8244003c20 con 0x7f82540b1730 2024-03-25T19:41:35.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.867+0000 7f82537fe640 10 monclient: get_auth_request con 0x7f824006e450 auth_method 0 2024-03-25T19:41:35.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.867+0000 7f8250ff9640 10 client.4236.objecter ms_handle_connect 0x7f824006e450 2024-03-25T19:41:35.878 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f8244054920 con 0x7f82540b1730 2024-03-25T19:41:35.878 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 10 client.4236.objecter ms_dispatch 0x7f8254000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:35.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 3 client.4236.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:35.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 3 client.4236.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:35.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 20 client.4236.objecter dump_active .. 0 homeless 2024-03-25T19:41:35.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8244050260 con 0x7f82540b1730 2024-03-25T19:41:35.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:35.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:35.878+0000 7f8250ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:36.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.135+0000 7f825a486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.135+0000 7f825a486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.135+0000 7f825a486640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:36.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.135+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82540627b0 con 0x7f82540b1730 2024-03-25T19:41:36.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.161+0000 7f8250ff9640 1 -- 172.21.15.50:0/3049581978 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f8244052360 con 0x7f82540b1730 2024-03-25T19:41:36.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.161+0000 7f8250ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:36.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.161+0000 7f8250ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:36.161 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:36.162 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 10 librados: watch_flush enter 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 10 librados: watch_flush exit 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 20 client.4236.objecter shutdown clearing up homeless session... 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 10 client.4236.objecter successfully canceled tick 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 >> v1:172.21.15.76:6800/2572556811 conn(0x7f824006e450 legacy=0x7f8240070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 10 monclient: shutdown 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:36.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 >> v1:172.21.15.50:6789/0 conn(0x7f82540b1730 legacy=0x7f825413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 shutdown_connections 2024-03-25T19:41:36.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 >> 172.21.15.50:0/3049581978 conn(0x7f82540617a0 msgr2=0x7f82540b6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:36.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 shutdown_connections 2024-03-25T19:41:36.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 1 -- 172.21.15.50:0/3049581978 wait complete. 2024-03-25T19:41:36.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:36.164+0000 7f825a486640 1 librados: shutdown 2024-03-25T19:41:36.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:37.845 INFO:tasks.ceph.osd.4.smithi076.stderr:2024-03-25T19:41:37.836+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:41:38.067 INFO:tasks.ceph.osd.5.smithi076.stderr:2024-03-25T19:41:38.064+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:41:38.225 INFO:tasks.ceph.osd.6.smithi076.stderr:2024-03-25T19:41:38.170+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:41:38.284 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-03-25T19:41:38.281+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:41:38.425 INFO:tasks.ceph.osd.7.smithi076.stderr:2024-03-25T19:41:38.422+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:41:38.923 INFO:tasks.ceph.osd.3.smithi050.stderr:2024-03-25T19:41:38.919+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:41:39.114 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-03-25T19:41:39.112+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:41:39.270 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-03-25T19:41:39.267+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:41:42.986 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:43.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.094+0000 7f583d0db640 10 monclient: get_monmap_and_config 2024-03-25T19:41:43.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.094+0000 7f583d0db640 10 monclient: build_initial_monmap 2024-03-25T19:41:43.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.095+0000 7f583d0db640 10 monclient: monmap: 2024-03-25T19:41:43.094 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:43.094 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:43.094 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:43.095819+0000 2024-03-25T19:41:43.095 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:43.095819+0000 2024-03-25T19:41:43.095 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:43.095 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:43.095 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:43.095 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:43.095 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:43.095 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.095+0000 7f583d0db640 1 Processor -- start 2024-03-25T19:41:43.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.095+0000 7f583d0db640 1 -- start start 2024-03-25T19:41:43.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.095+0000 7f583d0db640 10 monclient: init 2024-03-25T19:41:43.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:43.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 10 monclient(hunting): picked mon.noname-a con 0x7f58380b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 10 monclient(hunting): picked mon.noname-b con 0x7f58380b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 10 monclient(hunting): picked mon.noname-c con 0x7f583814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58380af940 con 0x7f58380b4d60 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58380b0a70 con 0x7f583814be90 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58380b0d50 con 0x7f58380b1730 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.096+0000 7f583d0db640 10 monclient(hunting): authenticate will time out at 4878.360041s 2024-03-25T19:41:43.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.120+0000 7f583759e640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f583814be90 0x7f583814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:47140/0 (socket says 172.21.15.50:47140) 2024-03-25T19:41:43.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.120+0000 7f583759e640 1 -- 172.21.15.50:0/2536109691 learned_addr learned my addr 172.21.15.50:0/2536109691 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:43.144 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.145+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940211711 0 0) 0x7f58380af940 con 0x7f58380b4d60 2024-03-25T19:41:43.144 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.145+0000 7f5835d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.145+0000 7f5835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.145+0000 7f5835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.145+0000 7f5835d9b640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:41:43.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.145+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5820003520 con 0x7f58380b4d60 2024-03-25T19:41:43.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.149+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050270778 0 0) 0x7f58380b0d50 con 0x7f58380b1730 2024-03-25T19:41:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.149+0000 7f5835d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.149+0000 7f5835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.149+0000 7f5835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.149+0000 7f5835d9b640 10 monclient(hunting): my global_id is 4244 2024-03-25T19:41:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.149+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58380af940 con 0x7f58380b1730 2024-03-25T19:41:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.151+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109460001 0 0) 0x7f58380b0a70 con 0x7f583814be90 2024-03-25T19:41:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.151+0000 7f5835d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.151+0000 7f5835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.151+0000 7f5835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.151+0000 7f5835d9b640 10 monclient(hunting): my global_id is 4243 2024-03-25T19:41:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.151+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58380b0d50 con 0x7f583814be90 2024-03-25T19:41:43.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865276347 0 0) 0x7f58380af940 con 0x7f58380b1730 2024-03-25T19:41:43.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 >> v1:172.21.15.76:6789/0 conn(0x7f583814be90 legacy=0x7f583814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 >> v1:172.21.15.50:6789/0 conn(0x7f58380b4d60 legacy=0x7f583813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 1 monclient: found mon.noname-b 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58380b1030 con 0x7f58380b1730 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f582c001670 con 0x7f58380b1730 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:43.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f5835d9b640 10 monclient: dump: 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f583d0db640 5 monclient: authenticate success, global_id 4244 2024-03-25T19:41:43.169 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.168+0000 7f583d0db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:43.184 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f582c0017d0 con 0x7f58380b1730 2024-03-25T19:41:43.184 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f5835d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:43.184 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f5835d9b640 1 -- 172.21.15.50:0/2536109691 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f582c001be0 con 0x7f58380b1730 2024-03-25T19:41:43.184 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f583d0db640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f583d0db640 4 set_mon_vals no callback set 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f5835d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f5835d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.185+0000 7f5835d9b640 10 monclient: dump: 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:43.185 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:43.186 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.186+0000 7f583d0db640 10 monclient: shutdown 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.186+0000 7f583d0db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.186+0000 7f583d0db640 1 -- 172.21.15.50:0/2536109691 >> v1:172.21.15.50:6790/0 conn(0x7f58380b1730 legacy=0x7f58380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.186+0000 7f583d0db640 1 -- 172.21.15.50:0/2536109691 shutdown_connections 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.186+0000 7f583d0db640 1 -- 172.21.15.50:0/2536109691 >> 172.21.15.50:0/2536109691 conn(0x7f58380617a0 msgr2=0x7f5838061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.186+0000 7f583d0db640 1 -- 172.21.15.50:0/2536109691 shutdown_connections 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.186+0000 7f583d0db640 1 -- 172.21.15.50:0/2536109691 wait complete. 2024-03-25T19:41:43.187 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 10 monclient: build_initial_monmap 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 10 monclient: monmap: 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:43.188 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 1 librados: starting msgr at 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 1 librados: starting objecter 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 1 Processor -- start 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 1 -- start start 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 1 librados: setting wanted keys 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 1 librados: calling monclient init 2024-03-25T19:41:43.189 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.187+0000 7f583d0db640 10 monclient: init 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 10 monclient(hunting): picked mon.noname-b con 0x7f58380b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 10 monclient(hunting): picked mon.noname-a con 0x7f58380b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 10 monclient(hunting): picked mon.noname-c con 0x7f583814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5838140860 con 0x7f58380b4d60 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58381385e0 con 0x7f58380b1730 2024-03-25T19:41:43.190 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5838137c70 con 0x7f583814be90 2024-03-25T19:41:43.191 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.191 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.188+0000 7f583d0db640 10 monclient(hunting): authenticate will time out at 4878.452397s 2024-03-25T19:41:43.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.194+0000 7f583759e640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f583814be90 0x7f58381dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:59984/0 (socket says 172.21.15.50:59984) 2024-03-25T19:41:43.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.194+0000 7f583759e640 1 -- 172.21.15.50:0/2402882724 learned_addr learned my addr 172.21.15.50:0/2402882724 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:43.196 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.197+0000 7f581f7fe640 10 client.?.objecter ms_handle_connect 0x7f583814be90 2024-03-25T19:41:43.196 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.197+0000 7f581f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.198+0000 7f581f7fe640 10 client.?.objecter ms_handle_connect 0x7f58380b1730 2024-03-25T19:41:43.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.198+0000 7f581f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.204 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.205+0000 7f581f7fe640 10 client.?.objecter ms_handle_connect 0x7f58380b4d60 2024-03-25T19:41:43.204 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.205+0000 7f581f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.209 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.209+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400351337 0 0) 0x7f58381385e0 con 0x7f58380b1730 2024-03-25T19:41:43.209 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.210+0000 7f581f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.209 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.210+0000 7f581f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.209 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.210+0000 7f581f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.209 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.210+0000 7f581f7fe640 10 monclient(hunting): my global_id is 4246 2024-03-25T19:41:43.209 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.210+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5824004eb0 con 0x7f58380b1730 2024-03-25T19:41:43.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.221+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950554461 0 0) 0x7f58380b1030 con 0x7f58380b4d60 2024-03-25T19:41:43.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.221+0000 7f581f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.221+0000 7f581f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.221+0000 7f581f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.221+0000 7f581f7fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:41:43.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.221+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58381385e0 con 0x7f58380b4d60 2024-03-25T19:41:43.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.227+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105880787 0 0) 0x7f5838137c70 con 0x7f583814be90 2024-03-25T19:41:43.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.227+0000 7f581f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.227+0000 7f581f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.227+0000 7f581f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.227+0000 7f581f7fe640 10 monclient(hunting): my global_id is 4247 2024-03-25T19:41:43.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.227+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58380b1030 con 0x7f583814be90 2024-03-25T19:41:43.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.235+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3699064484 0 0) 0x7f5824004eb0 con 0x7f58380b1730 2024-03-25T19:41:43.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.235+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5838137c70 con 0x7f58380b1730 2024-03-25T19:41:43.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.235+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5828001950 con 0x7f58380b1730 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.235+0000 7f581f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.235+0000 7f581f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.235+0000 7f581f7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.235+0000 7f581f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:43.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.240+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 586819379 0 0) 0x7f58381385e0 con 0x7f58380b4d60 2024-03-25T19:41:43.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.240+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5824004eb0 con 0x7f58380b4d60 2024-03-25T19:41:43.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.240+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f582c001940 con 0x7f58380b4d60 2024-03-25T19:41:43.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.240+0000 7f581f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.240+0000 7f581f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:43.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.240+0000 7f581f7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:43.239 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:43.240 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.243+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3660262776 0 0) 0x7f58380b1030 con 0x7f583814be90 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.243+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58381385e0 con 0x7f583814be90 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.243+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5830001bd0 con 0x7f583814be90 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.243+0000 7f581f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.243+0000 7f581f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.243+0000 7f581f7fe640 10 monclient(hunting): dump: 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:43.243 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.244 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.244 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.244 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:43.244 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:43.244 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:43.244 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:43.244 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2251425303 0 0) 0x7f5838137c70 con 0x7f58380b1730 2024-03-25T19:41:43.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 >> v1:172.21.15.50:6790/0 conn(0x7f583814be90 legacy=0x7f58381dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 >> v1:172.21.15.50:6789/0 conn(0x7f58380b4d60 legacy=0x7f5838132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 1 monclient: found mon.b 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838140aa0 con 0x7f58380b1730 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f581f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 5 monclient: authenticate success, global_id 4246 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 10 monclient: _renew_subs 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5838132750 con 0x7f58380b1730 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 10 monclient: _renew_subs 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5838144dd0 con 0x7f58380b1730 2024-03-25T19:41:43.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.253+0000 7f583d0db640 1 librados: init done 2024-03-25T19:41:43.255 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.254+0000 7f583d0db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.255 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.254+0000 7f583d0db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.255 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.254+0000 7f583d0db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:43.255 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.254+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5838132940 con 0x7f58380b1730 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.553+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5828003610 con 0x7f58380b1730 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.553+0000 7f581f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.553+0000 7f583559a640 4 set_mon_vals no callback set 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.553+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5828001d80 con 0x7f58380b1730 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.553+0000 7f581f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.553+0000 7f581f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.553+0000 7f581f7fe640 10 monclient: dump: 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:43.553 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:43.554 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:43.555 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.554+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f58280197d0 con 0x7f58380b1730 2024-03-25T19:41:43.555 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.555+0000 7f583659c640 10 monclient: get_auth_request con 0x7f582406e010 auth_method 0 2024-03-25T19:41:43.555 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.555+0000 7f581f7fe640 10 client.4246.objecter ms_handle_connect 0x7f582406e010 2024-03-25T19:41:43.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f58280534e0 con 0x7f58380b1730 2024-03-25T19:41:43.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 10 client.4246.objecter ms_dispatch 0x7f5838000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:43.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:43.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:43.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-25T19:41:43.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f582804ee20 con 0x7f58380b1730 2024-03-25T19:41:43.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.559+0000 7f581f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:43.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.818+0000 7f583d0db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:43.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.818+0000 7f583d0db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:43.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.818+0000 7f583d0db640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:43.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.818+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58380627b0 con 0x7f58380b1730 2024-03-25T19:41:43.842 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.843+0000 7f581f7fe640 1 -- 172.21.15.50:0/2402882724 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f5828050f20 con 0x7f58380b1730 2024-03-25T19:41:43.842 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.843+0000 7f581f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:43.842 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.843+0000 7f581f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:43.843 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:43.843 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:43.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 10 librados: watch_flush enter 2024-03-25T19:41:43.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 10 librados: watch_flush exit 2024-03-25T19:41:43.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-25T19:41:43.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 10 client.4246.objecter successfully canceled tick 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 >> v1:172.21.15.76:6800/2572556811 conn(0x7f582406e010 legacy=0x7f58240704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 10 monclient: shutdown 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 >> v1:172.21.15.76:6789/0 conn(0x7f58380b1730 legacy=0x7f5838137ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 shutdown_connections 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 >> 172.21.15.50:0/2402882724 conn(0x7f58380617a0 msgr2=0x7f58380b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 shutdown_connections 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.845+0000 7f583d0db640 1 -- 172.21.15.50:0/2402882724 wait complete. 2024-03-25T19:41:43.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:43.846+0000 7f583d0db640 1 librados: shutdown 2024-03-25T19:41:43.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:50.662 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:50.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.776+0000 7f4890ca5640 10 monclient: get_monmap_and_config 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.776+0000 7f4890ca5640 10 monclient: build_initial_monmap 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.776+0000 7f4890ca5640 10 monclient: monmap: 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:50.777433+0000 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:50.777433+0000 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:50.776 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:50.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.777+0000 7f4890ca5640 1 Processor -- start 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.777+0000 7f4890ca5640 1 -- start start 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.777+0000 7f4890ca5640 10 monclient: init 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 10 monclient(hunting): picked mon.noname-c con 0x7f488c0b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 10 monclient(hunting): picked mon.noname-b con 0x7f488c0b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 10 monclient(hunting): picked mon.noname-a con 0x7f488c14be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c0af940 con 0x7f488c14be90 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c0b0a70 con 0x7f488c0b4d60 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c0b0d50 con 0x7f488c0b1730 2024-03-25T19:41:50.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:50.779 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.778+0000 7f4890ca5640 10 monclient(hunting): authenticate will time out at 4886.041875s 2024-03-25T19:41:50.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.787+0000 7f488ad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f488c14be90 0x7f488c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53372/0 (socket says 172.21.15.50:53372) 2024-03-25T19:41:50.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.787+0000 7f488ad9d640 1 -- 172.21.15.50:0/3968528790 learned_addr learned my addr 172.21.15.50:0/3968528790 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:50.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.793+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361841876 0 0) 0x7f488c0af940 con 0x7f488c14be90 2024-03-25T19:41:50.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.793+0000 7f4889d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.793+0000 7f4889d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.793+0000 7f4889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.793+0000 7f4889d9b640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:41:50.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.793+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4878003520 con 0x7f488c14be90 2024-03-25T19:41:50.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946488925 0 0) 0x7f4878003520 con 0x7f488c14be90 2024-03-25T19:41:50.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 >> v1:172.21.15.50:6790/0 conn(0x7f488c0b1730 legacy=0x7f488c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 >> v1:172.21.15.76:6789/0 conn(0x7f488c0b4d60 legacy=0x7f488c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:50.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 1 monclient: found mon.noname-a 2024-03-25T19:41:50.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0b1030 con 0x7f488c14be90 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 10 monclient: _finish_auth 0 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4884001a10 con 0x7f488c14be90 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4890ca5640 5 monclient: authenticate success, global_id 4269 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4890ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4889d9b640 10 monclient: dump: 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:50.805 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:50.806 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.806 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:50.806 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:50.806 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:50.806 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:50.806 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:50.806 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.804+0000 7f4890ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:50.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4884001b70 con 0x7f488c14be90 2024-03-25T19:41:50.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4889d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:50.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4889d9b640 1 -- 172.21.15.50:0/3968528790 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4884001f80 con 0x7f488c14be90 2024-03-25T19:41:50.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4889d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4889d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4889d9b640 10 monclient: dump: 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4890ca5640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:50.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.815+0000 7f4890ca5640 4 set_mon_vals no callback set 2024-03-25T19:41:50.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.816+0000 7f4890ca5640 10 monclient: shutdown 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.816+0000 7f4890ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.816+0000 7f4890ca5640 1 -- 172.21.15.50:0/3968528790 >> v1:172.21.15.50:6789/0 conn(0x7f488c14be90 legacy=0x7f488c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.816+0000 7f4890ca5640 1 -- 172.21.15.50:0/3968528790 shutdown_connections 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.816+0000 7f4890ca5640 1 -- 172.21.15.50:0/3968528790 >> 172.21.15.50:0/3968528790 conn(0x7f488c0617a0 msgr2=0x7f488c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.816+0000 7f4890ca5640 1 -- 172.21.15.50:0/3968528790 shutdown_connections 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.816+0000 7f4890ca5640 1 -- 172.21.15.50:0/3968528790 wait complete. 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.817+0000 7f4890ca5640 10 monclient: build_initial_monmap 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.817+0000 7f4890ca5640 10 monclient: monmap: 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:50.817 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:50.818 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:50.818 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:50.818 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:50.818 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:50.818 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:50.818 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:50.818 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:50.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.818+0000 7f4890ca5640 1 librados: starting msgr at 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.818+0000 7f4890ca5640 1 librados: starting objecter 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.818+0000 7f4890ca5640 1 Processor -- start 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.818+0000 7f4890ca5640 1 -- start start 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.818+0000 7f4890ca5640 1 librados: setting wanted keys 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.818+0000 7f4890ca5640 1 librados: calling monclient init 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.818+0000 7f4890ca5640 10 monclient: init 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 10 monclient(hunting): picked mon.noname-a con 0x7f488c0b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 10 monclient(hunting): picked mon.noname-c con 0x7f488c0b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 10 monclient(hunting): picked mon.noname-b con 0x7f488c14be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:50.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c1307c0 con 0x7f488c0b1730 2024-03-25T19:41:50.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c138f90 con 0x7f488c14be90 2024-03-25T19:41:50.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c132ab0 con 0x7f488c0b4d60 2024-03-25T19:41:50.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:50.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.819+0000 7f4890ca5640 10 monclient(hunting): authenticate will time out at 4886.083030s 2024-03-25T19:41:50.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.829+0000 7f4883fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f488c0b4d60 0x7f488c12f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:59562/0 (socket says 172.21.15.50:59562) 2024-03-25T19:41:50.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.829+0000 7f4883fff640 1 -- 172.21.15.50:0/1019160144 learned_addr learned my addr 172.21.15.50:0/1019160144 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:50.830 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.831+0000 7f4882ffd640 10 client.?.objecter ms_handle_connect 0x7f488c0b4d60 2024-03-25T19:41:50.831 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.831+0000 7f4882ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.839+0000 7f4882ffd640 10 client.?.objecter ms_handle_connect 0x7f488c14be90 2024-03-25T19:41:50.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.839+0000 7f4882ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.843 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.843+0000 7f4882ffd640 10 client.?.objecter ms_handle_connect 0x7f488c0b1730 2024-03-25T19:41:50.843 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.843+0000 7f4882ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.853+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629535554 0 0) 0x7f488c138f90 con 0x7f488c14be90 2024-03-25T19:41:50.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.853+0000 7f4882ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.853+0000 7f4882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.853+0000 7f4882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.853+0000 7f4882ffd640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:41:50.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.853+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f486c004eb0 con 0x7f488c14be90 2024-03-25T19:41:50.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.859+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036619628 0 0) 0x7f488c1307c0 con 0x7f488c0b1730 2024-03-25T19:41:50.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.860+0000 7f4882ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.860+0000 7f4882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.860+0000 7f4882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.860+0000 7f4882ffd640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:41:50.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.860+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f488c138f90 con 0x7f488c0b1730 2024-03-25T19:41:50.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.864+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290039 0 0) 0x7f488c132ab0 con 0x7f488c0b4d60 2024-03-25T19:41:50.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.864+0000 7f4882ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.864+0000 7f4882ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.864+0000 7f4882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.864+0000 7f4882ffd640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:41:50.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.864+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f488c1307c0 con 0x7f488c0b4d60 2024-03-25T19:41:50.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.874+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2877618413 0 0) 0x7f488c138f90 con 0x7f488c0b1730 2024-03-25T19:41:50.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.874+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f488c132ab0 con 0x7f488c0b1730 2024-03-25T19:41:50.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.874+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f487c0018b0 con 0x7f488c0b1730 2024-03-25T19:41:50.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.874+0000 7f4882ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.874+0000 7f4882ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.874+0000 7f4882ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:50.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.874+0000 7f4882ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1233140707 0 0) 0x7f488c132ab0 con 0x7f488c0b1730 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 >> v1:172.21.15.50:6790/0 conn(0x7f488c0b4d60 legacy=0x7f488c12f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 >> v1:172.21.15.76:6789/0 conn(0x7f488c14be90 legacy=0x7f488c1300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 1 monclient: found mon.a 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c133a40 con 0x7f488c0b1730 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4882ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:50.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 10 monclient: _renew_subs 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c1316c0 con 0x7f488c0b1730 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 10 monclient: _renew_subs 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c137860 con 0x7f488c0b1730 2024-03-25T19:41:50.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.938+0000 7f4890ca5640 1 librados: init done 2024-03-25T19:41:50.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.939+0000 7f4890ca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:50.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.939+0000 7f4890ca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:50.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.939+0000 7f4890ca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:50.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:50.939+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c132c40 con 0x7f488c0b1730 2024-03-25T19:41:51.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.238+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f487c003610 con 0x7f488c0b1730 2024-03-25T19:41:51.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.239+0000 7f4882ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:51.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.239+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f487c003a60 con 0x7f488c0b1730 2024-03-25T19:41:51.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.239+0000 7f4882ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.239+0000 7f4882ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.239+0000 7f4882ffd640 10 monclient: dump: 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:51.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.239+0000 7f488959a640 4 set_mon_vals no callback set 2024-03-25T19:41:51.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.239+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f487c003c80 con 0x7f488c0b1730 2024-03-25T19:41:51.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.240+0000 7f4883fff640 10 monclient: get_auth_request con 0x7f486c06de20 auth_method 0 2024-03-25T19:41:51.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.240+0000 7f4882ffd640 10 client.4272.objecter ms_handle_connect 0x7f486c06de20 2024-03-25T19:41:51.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.246+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f487c053580 con 0x7f488c0b1730 2024-03-25T19:41:51.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.246+0000 7f4882ffd640 10 client.4272.objecter ms_dispatch 0x7f488c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:51.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.246+0000 7f4882ffd640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:51.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.246+0000 7f4882ffd640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:51.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.246+0000 7f4882ffd640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T19:41:51.247 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.247+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f487c04f090 con 0x7f488c0b1730 2024-03-25T19:41:51.247 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.247+0000 7f4882ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:51.247 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.247+0000 7f4882ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:51.498 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.499+0000 7f4890ca5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:51.498 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.499+0000 7f4890ca5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:51.498 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.499+0000 7f4890ca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:41:51.498 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.499+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f488c0627b0 con 0x7f488c0b1730 2024-03-25T19:41:51.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.518+0000 7f4882ffd640 1 -- 172.21.15.50:0/1019160144 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f487c050fc0 con 0x7f488c0b1730 2024-03-25T19:41:51.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.518+0000 7f4882ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:51.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.518+0000 7f4882ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:51.518 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:51.518 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 10 librados: watch_flush enter 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 10 librados: watch_flush exit 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 10 client.4272.objecter successfully canceled tick 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 >> v1:172.21.15.76:6800/2572556811 conn(0x7f486c06de20 legacy=0x7f486c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 10 monclient: shutdown 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 >> v1:172.21.15.50:6789/0 conn(0x7f488c0b1730 legacy=0x7f488c1393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:51.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 shutdown_connections 2024-03-25T19:41:51.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 >> 172.21.15.50:0/1019160144 conn(0x7f488c0617a0 msgr2=0x7f488c13c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:51.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 shutdown_connections 2024-03-25T19:41:51.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 1 -- 172.21.15.50:0/1019160144 wait complete. 2024-03-25T19:41:51.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:51.521+0000 7f4890ca5640 1 librados: shutdown 2024-03-25T19:41:51.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:54.007 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:14.514 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.070 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:14.576 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.095 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:14.566 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.101 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:14.607 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.154 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:14.625 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.164 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:14.670 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.321 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:14.827 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.396 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:14.866 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:54.417 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:14.922 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:58.339 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.453+0000 7f3d8cb3b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.453+0000 7f3d8cb3b640 10 monclient: build_initial_monmap 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.453+0000 7f3d8cb3b640 10 monclient: monmap: 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:41:58.454511+0000 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:41:58.454511+0000 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:58.453 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:58.454 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:41:58.454 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:41:58.454 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.454+0000 7f3d8cb3b640 1 Processor -- start 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.454+0000 7f3d8cb3b640 1 -- start start 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.454+0000 7f3d8cb3b640 10 monclient: init 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d880a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d8813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d881430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d88142520 con 0x7f3d880a98f0 2024-03-25T19:41:58.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d880b28a0 con 0x7f3d881430c0 2024-03-25T19:41:58.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d880b2a30 con 0x7f3d8813f710 2024-03-25T19:41:58.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:58.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.455+0000 7f3d8cb3b640 10 monclient(hunting): authenticate will time out at 4893.718967s 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.464+0000 7f3d8659c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f3d880a98f0 0x7f3d880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:57008/0 (socket says 172.21.15.50:57008) 2024-03-25T19:41:58.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.464+0000 7f3d8659c640 1 -- 172.21.15.50:0/2333017884 learned_addr learned my addr 172.21.15.50:0/2333017884 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410057329 0 0) 0x7f3d880b28a0 con 0x7f3d881430c0 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d70003520 con 0x7f3d881430c0 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86882445 0 0) 0x7f3d88142520 con 0x7f3d880a98f0 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:41:58.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.478+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d880b28a0 con 0x7f3d880a98f0 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069378100 0 0) 0x7f3d880b28a0 con 0x7f3d880a98f0 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 >> v1:172.21.15.50:6790/0 conn(0x7f3d8813f710 legacy=0x7f3d881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 >> v1:172.21.15.76:6789/0 conn(0x7f3d881430c0 legacy=0x7f3d8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 10 monclient: _finish_hunting 0 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 1 monclient: found mon.noname-a 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d880b44c0 con 0x7f3d880a98f0 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 10 monclient: _finish_auth 0 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 10 monclient: _check_auth_tickets 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3d78001670 con 0x7f3d880a98f0 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:58.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8559a640 10 monclient: dump: 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8cb3b640 5 monclient: authenticate success, global_id 4275 2024-03-25T19:41:58.492 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.491+0000 7f3d8cb3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d780017d0 con 0x7f3d880a98f0 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8cb3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8cb3b640 4 set_mon_vals no callback set 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8559a640 1 -- 172.21.15.50:0/2333017884 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3d78001ba0 con 0x7f3d880a98f0 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.502+0000 7f3d8559a640 10 monclient: dump: 2024-03-25T19:41:58.502 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:58.503 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:58.505 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.503+0000 7f3d8cb3b640 10 monclient: shutdown 2024-03-25T19:41:58.505 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.503+0000 7f3d8cb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.503+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2333017884 >> v1:172.21.15.50:6789/0 conn(0x7f3d880a98f0 legacy=0x7f3d880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.503+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2333017884 shutdown_connections 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.503+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2333017884 >> 172.21.15.50:0/2333017884 conn(0x7f3d880b7ea0 msgr2=0x7f3d880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.503+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2333017884 shutdown_connections 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.503+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2333017884 wait complete. 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.504+0000 7f3d8cb3b640 10 monclient: build_initial_monmap 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.504+0000 7f3d8cb3b640 10 monclient: monmap: 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:58.506 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 1 librados: starting msgr at 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 1 librados: starting objecter 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 1 Processor -- start 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 1 -- start start 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 1 librados: setting wanted keys 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 1 librados: calling monclient init 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 10 monclient: init 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d880a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:41:58.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d8813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:41:58.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.505+0000 7f3d8cb3b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d881430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:41:58.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.506+0000 7f3d8cb3b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d881398b0 con 0x7f3d881430c0 2024-03-25T19:41:58.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.506+0000 7f3d8cb3b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d880abb00 con 0x7f3d880a98f0 2024-03-25T19:41:58.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.506+0000 7f3d8cb3b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d880abde0 con 0x7f3d8813f710 2024-03-25T19:41:58.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.506+0000 7f3d8cb3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:58.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.506+0000 7f3d8cb3b640 10 monclient(hunting): authenticate will time out at 4893.769826s 2024-03-25T19:41:58.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.510+0000 7f3d8659c640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3d880a98f0 0x7f3d88136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:42004/0 (socket says 172.21.15.50:42004) 2024-03-25T19:41:58.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.510+0000 7f3d8659c640 1 -- 172.21.15.50:0/2991364682 learned_addr learned my addr 172.21.15.50:0/2991364682 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.515+0000 7f3d76ffd640 10 client.?.objecter ms_handle_connect 0x7f3d880a98f0 2024-03-25T19:41:58.514 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.515+0000 7f3d76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:58.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.518+0000 7f3d76ffd640 10 client.?.objecter ms_handle_connect 0x7f3d8813f710 2024-03-25T19:41:58.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.518+0000 7f3d76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:58.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.519+0000 7f3d76ffd640 10 client.?.objecter ms_handle_connect 0x7f3d881430c0 2024-03-25T19:41:58.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.519+0000 7f3d76ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:58.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.522+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046090254 0 0) 0x7f3d880abde0 con 0x7f3d8813f710 2024-03-25T19:41:58.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.523+0000 7f3d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.523+0000 7f3d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.523+0000 7f3d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.523+0000 7f3d76ffd640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:41:58.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.523+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d68004eb0 con 0x7f3d8813f710 2024-03-25T19:41:58.525 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.526+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182377097 0 0) 0x7f3d880abb00 con 0x7f3d880a98f0 2024-03-25T19:41:58.525 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.526+0000 7f3d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.525 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.526+0000 7f3d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.525 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.526+0000 7f3d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.525 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.526+0000 7f3d76ffd640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:41:58.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.526+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d880abde0 con 0x7f3d880a98f0 2024-03-25T19:41:58.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.539+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011454333 0 0) 0x7f3d881398b0 con 0x7f3d881430c0 2024-03-25T19:41:58.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.539+0000 7f3d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:58.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.539+0000 7f3d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:58.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.539+0000 7f3d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:58.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.539+0000 7f3d76ffd640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:41:58.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.539+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d880abb00 con 0x7f3d881430c0 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.540+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1068980409 0 0) 0x7f3d880abde0 con 0x7f3d880a98f0 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.540+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d881398b0 con 0x7f3d880a98f0 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.540+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3d78001ba0 con 0x7f3d880a98f0 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.540+0000 7f3d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.540+0000 7f3d76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.540+0000 7f3d76ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:58.540 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:58.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.540+0000 7f3d76ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.542+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3624877724 0 0) 0x7f3d68004eb0 con 0x7f3d8813f710 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.542+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d880abde0 con 0x7f3d8813f710 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.542+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3d7c0018c0 con 0x7f3d8813f710 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.542+0000 7f3d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.542+0000 7f3d76ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.542+0000 7f3d76ffd640 10 monclient(hunting): dump: 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.542 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.543 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.543 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:58.543 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:58.543 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:58.543 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:58.543 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4027630766 0 0) 0x7f3d881398b0 con 0x7f3d880a98f0 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 >> v1:172.21.15.50:6790/0 conn(0x7f3d8813f710 legacy=0x7f3d880ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 >> v1:172.21.15.50:6789/0 conn(0x7f3d881430c0 legacy=0x7f3d88138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 1 monclient: found mon.b 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88137af0 con 0x7f3d880a98f0 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:58.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 5 monclient: authenticate success, global_id 4258 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 10 monclient: _renew_subs 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d88138e10 con 0x7f3d880a98f0 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 10 monclient: _renew_subs 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d881e3480 con 0x7f3d880a98f0 2024-03-25T19:41:58.549 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.548+0000 7f3d8cb3b640 1 librados: init done 2024-03-25T19:41:58.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.549+0000 7f3d8cb3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:58.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.549+0000 7f3d8cb3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:58.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.549+0000 7f3d8cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:58.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.549+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d880b3ef0 con 0x7f3d880a98f0 2024-03-25T19:41:58.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.845+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d78004a90 con 0x7f3d880a98f0 2024-03-25T19:41:58.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.845+0000 7f3d76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:58.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.845+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3d78004da0 con 0x7f3d880a98f0 2024-03-25T19:41:58.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.845+0000 7f3d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:58.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.845+0000 7f3d76ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:58.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.845+0000 7f3d76ffd640 10 monclient: dump: 2024-03-25T19:41:58.845 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:41:58.845 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:41:58.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.845+0000 7f3d84d99640 4 set_mon_vals no callback set 2024-03-25T19:41:58.847 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.846+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f3d7801ad70 con 0x7f3d880a98f0 2024-03-25T19:41:58.847 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.846+0000 7f3d85d9b640 10 monclient: get_auth_request con 0x7f3d6806df70 auth_method 0 2024-03-25T19:41:58.847 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.847+0000 7f3d76ffd640 10 client.4258.objecter ms_handle_connect 0x7f3d6806df70 2024-03-25T19:41:58.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.851+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f3d78054920 con 0x7f3d880a98f0 2024-03-25T19:41:58.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.851+0000 7f3d76ffd640 10 client.4258.objecter ms_dispatch 0x7f3d88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:58.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.851+0000 7f3d76ffd640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:58.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.851+0000 7f3d76ffd640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:58.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.851+0000 7f3d76ffd640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T19:41:58.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.852+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d78050260 con 0x7f3d880a98f0 2024-03-25T19:41:58.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.852+0000 7f3d76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:58.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:58.852+0000 7f3d76ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:59.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.080+0000 7f3d8cb3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.080+0000 7f3d8cb3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.080+0000 7f3d8cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:41:59.080 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.080+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d880627b0 con 0x7f3d880a98f0 2024-03-25T19:41:59.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.125+0000 7f3d76ffd640 1 -- 172.21.15.50:0/2991364682 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f3d78052360 con 0x7f3d880a98f0 2024-03-25T19:41:59.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.125+0000 7f3d76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:59.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.125+0000 7f3d76ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:59.125 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:41:59.125 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:59.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 10 librados: watch_flush enter 2024-03-25T19:41:59.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 10 librados: watch_flush exit 2024-03-25T19:41:59.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T19:41:59.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 10 client.4258.objecter successfully canceled tick 2024-03-25T19:41:59.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 >> v1:172.21.15.76:6800/2572556811 conn(0x7f3d6806df70 legacy=0x7f3d68070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 10 monclient: shutdown 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 >> v1:172.21.15.76:6789/0 conn(0x7f3d880a98f0 legacy=0x7f3d88136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 shutdown_connections 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 >> 172.21.15.50:0/2991364682 conn(0x7f3d880b7ea0 msgr2=0x7f3d88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 shutdown_connections 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 1 -- 172.21.15.50:0/2991364682 wait complete. 2024-03-25T19:41:59.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:41:59.127+0000 7f3d8cb3b640 1 librados: shutdown 2024-03-25T19:41:59.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:05.943 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.057+0000 7fb5dd33d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.057+0000 7fb5dd33d640 10 monclient: build_initial_monmap 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.057+0000 7fb5dd33d640 10 monclient: monmap: 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:06.058720+0000 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:06.058720+0000 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:06.057 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:06.058 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:06.058 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:06.058 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:06.058 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:06.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.058+0000 7fb5dd33d640 1 Processor -- start 2024-03-25T19:42:06.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.058+0000 7fb5dd33d640 1 -- start start 2024-03-25T19:42:06.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.058+0000 7fb5dd33d640 10 monclient: init 2024-03-25T19:42:06.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:06.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:06.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 10 monclient(hunting): picked mon.noname-b con 0x7fb5d80b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:06.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 10 monclient(hunting): picked mon.noname-c con 0x7fb5d80b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:06.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5d814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:06.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d80af940 con 0x7fb5d814be90 2024-03-25T19:42:06.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d80b0a70 con 0x7fb5d80b1730 2024-03-25T19:42:06.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d80b0d50 con 0x7fb5d80b4d60 2024-03-25T19:42:06.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:06.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.059+0000 7fb5dd33d640 10 monclient(hunting): authenticate will time out at 4901.323150s 2024-03-25T19:42:06.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.072+0000 7fb5d77fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fb5d814be90 0x7fb5d814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33044/0 (socket says 172.21.15.50:33044) 2024-03-25T19:42:06.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.073+0000 7fb5d77fe640 1 -- 172.21.15.50:0/2676517151 learned_addr learned my addr 172.21.15.50:0/2676517151 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:06.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.078+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043482395 0 0) 0x7fb5d80b0a70 con 0x7fb5d80b1730 2024-03-25T19:42:06.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.078+0000 7fb5d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.078+0000 7fb5d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.078+0000 7fb5d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.078+0000 7fb5d67fc640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:42:06.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.078+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c4003520 con 0x7fb5d80b1730 2024-03-25T19:42:06.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.087+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 838060383 0 0) 0x7fb5c4003520 con 0x7fb5d80b1730 2024-03-25T19:42:06.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.087+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 >> v1:172.21.15.50:6790/0 conn(0x7fb5d80b4d60 legacy=0x7fb5d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.087+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 >> v1:172.21.15.50:6789/0 conn(0x7fb5d814be90 legacy=0x7fb5d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.087+0000 7fb5d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:06.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.087+0000 7fb5d67fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d80b1030 con 0x7fb5d80b1730 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb5c0001690 con 0x7fb5d80b1730 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5d67fc640 10 monclient: dump: 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.088 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5dd33d640 5 monclient: authenticate success, global_id 4261 2024-03-25T19:42:06.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.088+0000 7fb5dd33d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5c00017f0 con 0x7fb5d80b1730 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5d67fc640 1 -- 172.21.15.50:0/2676517151 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb5c0001c00 con 0x7fb5d80b1730 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5d67fc640 10 monclient: dump: 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:06.108 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:06.109 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:06.109 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:06.109 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:06.109 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5dd33d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:06.109 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.108+0000 7fb5dd33d640 4 set_mon_vals no callback set 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.109+0000 7fb5dd33d640 10 monclient: shutdown 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.109+0000 7fb5dd33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.109+0000 7fb5dd33d640 1 -- 172.21.15.50:0/2676517151 >> v1:172.21.15.76:6789/0 conn(0x7fb5d80b1730 legacy=0x7fb5d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.109+0000 7fb5dd33d640 1 -- 172.21.15.50:0/2676517151 shutdown_connections 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.109+0000 7fb5dd33d640 1 -- 172.21.15.50:0/2676517151 >> 172.21.15.50:0/2676517151 conn(0x7fb5d80617a0 msgr2=0x7fb5d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.109+0000 7fb5dd33d640 1 -- 172.21.15.50:0/2676517151 shutdown_connections 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.109+0000 7fb5dd33d640 1 -- 172.21.15.50:0/2676517151 wait complete. 2024-03-25T19:42:06.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 10 monclient: build_initial_monmap 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 10 monclient: monmap: 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:06.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 1 librados: starting msgr at 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 1 librados: starting objecter 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 1 Processor -- start 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 1 -- start start 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 1 librados: setting wanted keys 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 1 librados: calling monclient init 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 10 monclient: init 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 10 monclient(hunting): picked mon.noname-c con 0x7fb5d80b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.110+0000 7fb5dd33d640 10 monclient(hunting): picked mon.noname-a con 0x7fb5d80b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.111+0000 7fb5dd33d640 10 monclient(hunting): picked mon.noname-b con 0x7fb5d814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.111+0000 7fb5dd33d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d81d83d0 con 0x7fb5d80b4d60 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.111+0000 7fb5dd33d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d81d77a0 con 0x7fb5d814be90 2024-03-25T19:42:06.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.111+0000 7fb5dd33d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d8138270 con 0x7fb5d80b1730 2024-03-25T19:42:06.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.111+0000 7fb5dd33d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:06.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.111+0000 7fb5dd33d640 10 monclient(hunting): authenticate will time out at 4901.374787s 2024-03-25T19:42:06.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.118+0000 7fb5d6ffd640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fb5d80b1730 0x7fb5d81d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:34056/0 (socket says 172.21.15.50:34056) 2024-03-25T19:42:06.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.118+0000 7fb5d6ffd640 1 -- 172.21.15.50:0/929970410 learned_addr learned my addr 172.21.15.50:0/929970410 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:06.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.121+0000 7fb5cf7fe640 10 client.?.objecter ms_handle_connect 0x7fb5d80b1730 2024-03-25T19:42:06.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.121+0000 7fb5cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.134+0000 7fb5cf7fe640 10 client.?.objecter ms_handle_connect 0x7fb5d814be90 2024-03-25T19:42:06.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.134+0000 7fb5cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.138 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.139+0000 7fb5cf7fe640 10 client.?.objecter ms_handle_connect 0x7fb5d80b4d60 2024-03-25T19:42:06.138 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.139+0000 7fb5cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267957445 0 0) 0x7fb5d81d77a0 con 0x7fb5d814be90 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b8004eb0 con 0x7fb5d814be90 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687918991 0 0) 0x7fb5d8138270 con 0x7fb5d80b1730 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:42:06.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.143+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d81d77a0 con 0x7fb5d80b1730 2024-03-25T19:42:06.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.151+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1472745413 0 0) 0x7fb5b8004eb0 con 0x7fb5d814be90 2024-03-25T19:42:06.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.151+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5d8138270 con 0x7fb5d814be90 2024-03-25T19:42:06.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.151+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb5d0001b00 con 0x7fb5d814be90 2024-03-25T19:42:06.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.151+0000 7fb5cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.151+0000 7fb5cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.151+0000 7fb5cf7fe640 10 monclient(hunting): dump: 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:06.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.151+0000 7fb5cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:06.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.153+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992619119 0 0) 0x7fb5d81d83d0 con 0x7fb5d80b4d60 2024-03-25T19:42:06.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.153+0000 7fb5cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:06.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.153+0000 7fb5cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:06.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.153+0000 7fb5cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:06.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.153+0000 7fb5cf7fe640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:42:06.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.153+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b8004eb0 con 0x7fb5d80b4d60 2024-03-25T19:42:06.169 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1176026803 0 0) 0x7fb5d8138270 con 0x7fb5d814be90 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 >> v1:172.21.15.50:6790/0 conn(0x7fb5d80b1730 legacy=0x7fb5d81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 >> v1:172.21.15.50:6789/0 conn(0x7fb5d80b4d60 legacy=0x7fb5d81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 1 monclient: found mon.b 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d8140d00 con 0x7fb5d814be90 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:42:06.170 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 10 monclient: _renew_subs 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d8138a80 con 0x7fb5d814be90 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 10 monclient: _renew_subs 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d8143640 con 0x7fb5d814be90 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.170+0000 7fb5dd33d640 1 librados: init done 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.171+0000 7fb5dd33d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.171 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.171+0000 7fb5dd33d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.172 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.171+0000 7fb5dd33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:06.172 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.171+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d8140ef0 con 0x7fb5d814be90 2024-03-25T19:42:06.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.454+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5d0001f70 con 0x7fb5d814be90 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.454+0000 7fb5cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.454+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb5d0004b60 con 0x7fb5d814be90 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.454+0000 7fb5cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.454+0000 7fb5cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.454+0000 7fb5cf7fe640 10 monclient: dump: 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:06.454 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:06.455 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:06.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.454+0000 7fb5d5ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:06.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.455+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fb5d0004d80 con 0x7fb5d814be90 2024-03-25T19:42:06.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.455+0000 7fb5d6ffd640 10 monclient: get_auth_request con 0x7fb5b806de20 auth_method 0 2024-03-25T19:42:06.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.456+0000 7fb5cf7fe640 10 client.4264.objecter ms_handle_connect 0x7fb5b806de20 2024-03-25T19:42:06.458 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.459+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fb5d0054760 con 0x7fb5d814be90 2024-03-25T19:42:06.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.459+0000 7fb5cf7fe640 10 client.4264.objecter ms_dispatch 0x7fb5d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:06.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.459+0000 7fb5cf7fe640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:06.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.459+0000 7fb5cf7fe640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:06.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.459+0000 7fb5cf7fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T19:42:06.460 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.461+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5d0050270 con 0x7fb5d814be90 2024-03-25T19:42:06.460 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.461+0000 7fb5cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.460 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.461+0000 7fb5cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:06.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.709+0000 7fb5dd33d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.709+0000 7fb5dd33d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.709+0000 7fb5dd33d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:06.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.709+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5d80afbd0 con 0x7fb5d814be90 2024-03-25T19:42:06.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.734+0000 7fb5cf7fe640 1 -- 172.21.15.50:0/929970410 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fb5d00521a0 con 0x7fb5d814be90 2024-03-25T19:42:06.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.734+0000 7fb5cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.734+0000 7fb5cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:06.735 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:42:06.735 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 10 librados: watch_flush enter 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 10 librados: watch_flush exit 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 10 client.4264.objecter successfully canceled tick 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 >> v1:172.21.15.76:6800/2572556811 conn(0x7fb5b806de20 legacy=0x7fb5b80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 10 monclient: shutdown 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 >> v1:172.21.15.76:6789/0 conn(0x7fb5d814be90 legacy=0x7fb5d8142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 shutdown_connections 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 >> 172.21.15.50:0/929970410 conn(0x7fb5d80617a0 msgr2=0x7fb5d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 shutdown_connections 2024-03-25T19:42:06.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 1 -- 172.21.15.50:0/929970410 wait complete. 2024-03-25T19:42:06.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:06.737+0000 7fb5dd33d640 1 librados: shutdown 2024-03-25T19:42:06.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:13.556 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:13.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.673+0000 7f49b72f1640 10 monclient: get_monmap_and_config 2024-03-25T19:42:13.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.673+0000 7f49b72f1640 10 monclient: build_initial_monmap 2024-03-25T19:42:13.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.673+0000 7f49b72f1640 10 monclient: monmap: 2024-03-25T19:42:13.853 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:13.853 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:13.853 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:13.675279+0000 2024-03-25T19:42:13.853 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:13.675279+0000 2024-03-25T19:42:13.854 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:13.854 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:13.854 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:13.854 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:13.854 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:13.854 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.674+0000 7f49b72f1640 1 Processor -- start 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.674+0000 7f49b72f1640 1 -- start start 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.674+0000 7f49b72f1640 10 monclient: init 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.674+0000 7f49b72f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.674+0000 7f49b72f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.674+0000 7f49b72f1640 10 monclient(hunting): picked mon.noname-c con 0x7f49b00b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.674+0000 7f49b72f1640 10 monclient(hunting): picked mon.noname-b con 0x7f49b00b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.675+0000 7f49b72f1640 10 monclient(hunting): picked mon.noname-a con 0x7f49b014be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.675+0000 7f49b72f1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00af940 con 0x7f49b014be90 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.675+0000 7f49b72f1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00b0a70 con 0x7f49b00b4d60 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.675+0000 7f49b72f1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b00b0d50 con 0x7f49b00b1730 2024-03-25T19:42:13.862 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.675+0000 7f49b72f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.675+0000 7f49b72f1640 10 monclient(hunting): authenticate will time out at 4908.939744s 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.684+0000 7f49b508d640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f49b00b4d60 0x7f49b013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:58308/0 (socket says 172.21.15.50:58308) 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.684+0000 7f49b508d640 1 -- 172.21.15.50:0/2034432951 learned_addr learned my addr 172.21.15.50:0/2034432951 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.698+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527823998 0 0) 0x7f49b00b0a70 con 0x7f49b00b4d60 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.698+0000 7f49a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.698+0000 7f49a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.698+0000 7f49a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.698+0000 7f49a7fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.698+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f499c003520 con 0x7f49b00b4d60 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.711+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762988445 0 0) 0x7f49b00b0d50 con 0x7f49b00b1730 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.711+0000 7f49a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.711+0000 7f49a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.711+0000 7f49a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.711+0000 7f49a7fff640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:42:13.863 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.711+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b00b0a70 con 0x7f49b00b1730 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.713+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646247424 0 0) 0x7f49b00af940 con 0x7f49b014be90 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.713+0000 7f49a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.713+0000 7f49a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.713+0000 7f49a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.713+0000 7f49a7fff640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.713+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b00b0d50 con 0x7f49b014be90 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239691140 0 0) 0x7f499c003520 con 0x7f49b00b4d60 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 >> v1:172.21.15.50:6790/0 conn(0x7f49b00b1730 legacy=0x7f49b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 >> v1:172.21.15.50:6789/0 conn(0x7f49b014be90 legacy=0x7f49b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 1 monclient: found mon.noname-c 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:42:13.864 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b00b1030 con 0x7f49b00b4d60 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f49ac001690 con 0x7f49b00b4d60 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49a7fff640 10 monclient: dump: 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:13.865 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49b72f1640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.721+0000 7f49b72f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49ac0017f0 con 0x7f49b00b4d60 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49a7fff640 1 -- 172.21.15.50:0/2034432951 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f49ac001c00 con 0x7f49b00b4d60 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49b72f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49b72f1640 4 set_mon_vals no callback set 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.742+0000 7f49a7fff640 10 monclient: dump: 2024-03-25T19:42:13.866 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.743+0000 7f49b72f1640 10 monclient: shutdown 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.743+0000 7f49b72f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.743+0000 7f49b72f1640 1 -- 172.21.15.50:0/2034432951 >> v1:172.21.15.76:6789/0 conn(0x7f49b00b4d60 legacy=0x7f49b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.743+0000 7f49b72f1640 1 -- 172.21.15.50:0/2034432951 shutdown_connections 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.743+0000 7f49b72f1640 1 -- 172.21.15.50:0/2034432951 >> 172.21.15.50:0/2034432951 conn(0x7f49b00617a0 msgr2=0x7f49b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.743+0000 7f49b72f1640 1 -- 172.21.15.50:0/2034432951 shutdown_connections 2024-03-25T19:42:13.867 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.743+0000 7f49b72f1640 1 -- 172.21.15.50:0/2034432951 wait complete. 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 10 monclient: build_initial_monmap 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 10 monclient: monmap: 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 1 librados: starting msgr at 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 1 librados: starting objecter 2024-03-25T19:42:13.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 1 Processor -- start 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 1 -- start start 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 1 librados: setting wanted keys 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 1 librados: calling monclient init 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.744+0000 7f49b72f1640 10 monclient: init 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.745+0000 7f49b72f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.745+0000 7f49b72f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.745+0000 7f49b72f1640 10 monclient(hunting): picked mon.noname-b con 0x7f49b00b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.746+0000 7f49b72f1640 10 monclient(hunting): picked mon.noname-c con 0x7f49b00b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.746+0000 7f49b72f1640 10 monclient(hunting): picked mon.noname-a con 0x7f49b014be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.746+0000 7f49b72f1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b0141ce0 con 0x7f49b014be90 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.746+0000 7f49b72f1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b0142610 con 0x7f49b00b1730 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.746+0000 7f49b72f1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b01dae80 con 0x7f49b00b4d60 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.746+0000 7f49b72f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.746+0000 7f49b72f1640 10 monclient(hunting): authenticate will time out at 4909.010884s 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.753+0000 7f49b588e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f49b014be90 0x7f49b0140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51826/0 (socket says 172.21.15.50:51826) 2024-03-25T19:42:13.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.753+0000 7f49b588e640 1 -- 172.21.15.50:0/1961375430 learned_addr learned my addr 172.21.15.50:0/1961375430 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.754+0000 7f49a5ffb640 10 client.?.objecter ms_handle_connect 0x7f49b014be90 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.754+0000 7f49a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.761+0000 7f49a5ffb640 10 client.?.objecter ms_handle_connect 0x7f49b00b4d60 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.761+0000 7f49a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.767+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647686341 0 0) 0x7f49b0141ce0 con 0x7f49b014be90 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.767+0000 7f49a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.767+0000 7f49a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.767+0000 7f49a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.767+0000 7f49a5ffb640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.767+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4994004eb0 con 0x7f49b014be90 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 10 client.?.objecter ms_handle_connect 0x7f49b00b1730 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213593977 0 0) 0x7f49b01dae80 con 0x7f49b00b4d60 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:13.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.772+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b0141ce0 con 0x7f49b00b4d60 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.781+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727836392 0 0) 0x7f49b0142610 con 0x7f49b00b1730 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.781+0000 7f49a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.781+0000 7f49a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.781+0000 7f49a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.781+0000 7f49a5ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.782+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b01dae80 con 0x7f49b00b1730 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.783+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 615971566 0 0) 0x7f4994004eb0 con 0x7f49b014be90 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.784+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49b0142610 con 0x7f49b014be90 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.784+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f49a8001c70 con 0x7f49b014be90 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.784+0000 7f49a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.784+0000 7f49a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.784+0000 7f49a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:13.871 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.784+0000 7f49a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.789+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681430163 0 0) 0x7f49b0141ce0 con 0x7f49b00b4d60 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.789+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4994004eb0 con 0x7f49b00b4d60 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.789+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f49a0001970 con 0x7f49b00b4d60 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.789+0000 7f49a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.789+0000 7f49a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.789+0000 7f49a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:13.872 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.798+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3668976399 0 0) 0x7f49b01dae80 con 0x7f49b00b1730 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.798+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49b0141ce0 con 0x7f49b00b1730 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.798+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f49ac001b00 con 0x7f49b00b1730 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.798+0000 7f49a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.798+0000 7f49a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.798+0000 7f49a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:13.873 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1282766338 0 0) 0x7f4994004eb0 con 0x7f49b00b4d60 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 >> v1:172.21.15.76:6789/0 conn(0x7f49b00b1730 legacy=0x7f49b0142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 >> v1:172.21.15.50:6789/0 conn(0x7f49b014be90 legacy=0x7f49b0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 1 monclient: found mon.c 2024-03-25T19:42:13.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b01db010 con 0x7f49b00b4d60 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49b72f1640 5 monclient: authenticate success, global_id 4271 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49b72f1640 10 monclient: _renew_subs 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49b72f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.799+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b0143240 con 0x7f49b00b4d60 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.800+0000 7f49b72f1640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.800+0000 7f49b72f1640 10 monclient: _renew_subs 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.800+0000 7f49b72f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.800+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b013bd20 con 0x7f49b00b4d60 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.800+0000 7f49b72f1640 1 librados: init done 2024-03-25T19:42:13.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.801+0000 7f49b72f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:13.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.801+0000 7f49b72f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:13.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.801+0000 7f49b72f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:42:13.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:13.801+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b01427a0 con 0x7f49b00b4d60 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.116+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49a00035f0 con 0x7f49b00b4d60 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.116+0000 7f49a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.116+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f49a0003900 con 0x7f49b00b4d60 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.116+0000 7f49a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.116+0000 7f49a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.116+0000 7f49a5ffb640 10 monclient: dump: 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:14.117 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:14.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.116+0000 7f49a77fe640 4 set_mon_vals no callback set 2024-03-25T19:42:14.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.117+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f49a0019910 con 0x7f49b00b4d60 2024-03-25T19:42:14.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.118+0000 7f49b508d640 10 monclient: get_auth_request con 0x7f499406e180 auth_method 0 2024-03-25T19:42:14.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.118+0000 7f49a5ffb640 10 client.4271.objecter ms_handle_connect 0x7f499406e180 2024-03-25T19:42:14.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f49a0053580 con 0x7f49b00b4d60 2024-03-25T19:42:14.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 10 client.4271.objecter ms_dispatch 0x7f49b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:14.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:14.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:14.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-25T19:42:14.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49a004eec0 con 0x7f49b00b4d60 2024-03-25T19:42:14.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.124+0000 7f49a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:14.187 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:34.694 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.197 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:34.704 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.260 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:34.766 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.266 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:34.736 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.272 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:34.742 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.280 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:34.786 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.290 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:34.796 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.326 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:34.797 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.339+0000 7f49b72f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.339+0000 7f49b72f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.339+0000 7f49b72f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:42:14.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.339+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49b00627b0 con 0x7f49b00b4d60 2024-03-25T19:42:14.365 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:34.871 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:14.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.385+0000 7f49a5ffb640 1 -- 172.21.15.50:0/1961375430 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f49a0050fc0 con 0x7f49b00b4d60 2024-03-25T19:42:14.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.385+0000 7f49a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.385+0000 7f49a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:14.386 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:42:14.387 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:14.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 10 librados: watch_flush enter 2024-03-25T19:42:14.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 10 librados: watch_flush exit 2024-03-25T19:42:14.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-25T19:42:14.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 10 client.4271.objecter successfully canceled tick 2024-03-25T19:42:14.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 >> v1:172.21.15.76:6800/2572556811 conn(0x7f499406e180 legacy=0x7f4994070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 10 monclient: shutdown 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 >> v1:172.21.15.50:6790/0 conn(0x7f49b00b4d60 legacy=0x7f49b01db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 shutdown_connections 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 >> 172.21.15.50:0/1961375430 conn(0x7f49b00617a0 msgr2=0x7f49b00b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 shutdown_connections 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 1 -- 172.21.15.50:0/1961375430 wait complete. 2024-03-25T19:42:14.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:14.388+0000 7f49b72f1640 1 librados: shutdown 2024-03-25T19:42:14.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:21.205 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:21.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.322+0000 7f7dd5daf640 10 monclient: get_monmap_and_config 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.323+0000 7f7dd5daf640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.323+0000 7f7dd5daf640 10 monclient: monmap: 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:21.323843+0000 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:21.323843+0000 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:21.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.323+0000 7f7dd5daf640 1 Processor -- start 2024-03-25T19:42:21.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.323+0000 7f7dd5daf640 1 -- start start 2024-03-25T19:42:21.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.323+0000 7f7dd5daf640 10 monclient: init 2024-03-25T19:42:21.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 10 monclient(hunting): picked mon.noname-b con 0x7f7dd00a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 10 monclient(hunting): picked mon.noname-c con 0x7f7dd013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 10 monclient(hunting): picked mon.noname-a con 0x7f7dd01430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd0142520 con 0x7f7dd01430c0 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd00b28a0 con 0x7f7dd013f710 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd00b2a30 con 0x7f7dd00a98f0 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.324+0000 7f7dd5daf640 10 monclient(hunting): authenticate will time out at 4916.588569s 2024-03-25T19:42:21.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.336+0000 7f7dcf7fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f7dd00a98f0 0x7f7dd00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:60536/0 (socket says 172.21.15.50:60536) 2024-03-25T19:42:21.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.336+0000 7f7dcf7fe640 1 -- 172.21.15.50:0/2638764759 learned_addr learned my addr 172.21.15.50:0/2638764759 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:21.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117991016 0 0) 0x7f7dd00b28a0 con 0x7f7dd013f710 2024-03-25T19:42:21.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db8003520 con 0x7f7dd013f710 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506229887 0 0) 0x7f7dd0142520 con 0x7f7dd01430c0 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:42:21.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.351+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd00b28a0 con 0x7f7dd01430c0 2024-03-25T19:42:21.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.362+0000 7f7dcf7fe640 1 -- 172.21.15.50:0/2638764759 >> v1:172.21.15.50:6790/0 conn(0x7f7dd00a98f0 legacy=0x7f7dd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:42:21.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.362+0000 7f7dcf7fe640 1 -- 172.21.15.50:0/2638764759 >> v1:172.21.15.50:6790/0 conn(0x7f7dd00a98f0 legacy=0x7f7dd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:42:21.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.362+0000 7f7dcf7fe640 1 --1- 172.21.15.50:0/2638764759 >> v1:172.21.15.50:6790/0 conn(0x7f7dd00a98f0 0x7f7dd00a9cd0 :-1 s=OPENED pgs=5377 cs=1 l=1).handle_message read tag failed 2024-03-25T19:42:21.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.362+0000 7f7dcf7fe640 1 --1- 172.21.15.50:0/2638764759 >> v1:172.21.15.50:6790/0 conn(0x7f7dd00a98f0 0x7f7dd00a9cd0 :-1 s=OPENED pgs=5377 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:42:21.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.362+0000 7f7dce7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.50:6790/0 2024-03-25T19:42:21.364 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988102451 0 0) 0x7f7dd00b28a0 con 0x7f7dd01430c0 2024-03-25T19:42:21.364 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 >> v1:172.21.15.50:6790/0 conn(0x7f7dd00a98f0 legacy=0x7f7dd00a9cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:42:21.364 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 >> v1:172.21.15.76:6789/0 conn(0x7f7dd013f710 legacy=0x7f7dd01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.364 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:21.364 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 1 monclient: found mon.noname-a 2024-03-25T19:42:21.364 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd00b44c0 con 0x7f7dd01430c0 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7dc80019f0 con 0x7f7dd01430c0 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dd5daf640 5 monclient: authenticate success, global_id 4293 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dd5daf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dce7fc640 10 monclient: dump: 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:21.365 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:21.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.364+0000 7f7dd5daf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:21.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dc8001b50 con 0x7f7dd01430c0 2024-03-25T19:42:21.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dce7fc640 1 -- 172.21.15.50:0/2638764759 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7dc8001fa0 con 0x7f7dd01430c0 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dce7fc640 10 monclient: dump: 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:21.386 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:21.387 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:21.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dd5daf640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:21.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.386+0000 7f7dd5daf640 4 set_mon_vals no callback set 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.387+0000 7f7dd5daf640 10 monclient: shutdown 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.387+0000 7f7dd5daf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.387+0000 7f7dd5daf640 1 -- 172.21.15.50:0/2638764759 >> v1:172.21.15.50:6789/0 conn(0x7f7dd01430c0 legacy=0x7f7dd014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.387+0000 7f7dd5daf640 1 -- 172.21.15.50:0/2638764759 shutdown_connections 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.387+0000 7f7dd5daf640 1 -- 172.21.15.50:0/2638764759 >> 172.21.15.50:0/2638764759 conn(0x7f7dd00b7ea0 msgr2=0x7f7dd00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.387+0000 7f7dd5daf640 1 -- 172.21.15.50:0/2638764759 shutdown_connections 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.387+0000 7f7dd5daf640 1 -- 172.21.15.50:0/2638764759 wait complete. 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 10 monclient: monmap: 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:21.389 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 1 librados: starting msgr at 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 1 librados: starting objecter 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 1 Processor -- start 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 1 -- start start 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 1 librados: setting wanted keys 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 1 librados: calling monclient init 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.388+0000 7f7dd5daf640 10 monclient: init 2024-03-25T19:42:21.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 10 monclient(hunting): picked mon.noname-a con 0x7f7dd00a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 10 monclient(hunting): picked mon.noname-b con 0x7f7dd013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 10 monclient(hunting): picked mon.noname-c con 0x7f7dd01430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd00ab9e0 con 0x7f7dd00a98f0 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd00abcc0 con 0x7f7dd013f710 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd00abfa0 con 0x7f7dd01430c0 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.389+0000 7f7dd5daf640 10 monclient(hunting): authenticate will time out at 4916.653202s 2024-03-25T19:42:21.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.400+0000 7f7dcf7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7dd00a98f0 0x7f7dd01387d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56418/0 (socket says 172.21.15.50:56418) 2024-03-25T19:42:21.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.400+0000 7f7dcf7fe640 1 -- 172.21.15.50:0/1405051820 learned_addr learned my addr 172.21.15.50:0/1405051820 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:21.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.402+0000 7f7db7fff640 10 client.?.objecter ms_handle_connect 0x7f7dd013f710 2024-03-25T19:42:21.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.402+0000 7f7db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.407+0000 7f7db7fff640 10 client.?.objecter ms_handle_connect 0x7f7dd00a98f0 2024-03-25T19:42:21.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.407+0000 7f7db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.408 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.408+0000 7f7db7fff640 10 client.?.objecter ms_handle_connect 0x7f7dd01430c0 2024-03-25T19:42:21.408 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.408+0000 7f7db7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063514735 0 0) 0x7f7dd00abcc0 con 0x7f7dd013f710 2024-03-25T19:42:21.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:42:21.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dbc004eb0 con 0x7f7dd013f710 2024-03-25T19:42:21.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330252806 0 0) 0x7f7dd00abfa0 con 0x7f7dd01430c0 2024-03-25T19:42:21.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:42:21.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.415+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dd00abcc0 con 0x7f7dd01430c0 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.426+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352705897 0 0) 0x7f7dbc004eb0 con 0x7f7dd013f710 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.426+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dd00abfa0 con 0x7f7dd013f710 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.426+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7dc4001950 con 0x7f7dd013f710 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.426+0000 7f7db7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.426+0000 7f7db7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.426+0000 7f7db7fff640 10 monclient(hunting): dump: 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:21.425 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:21.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.426+0000 7f7db7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:21.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.427+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591931821 0 0) 0x7f7dd00ab9e0 con 0x7f7dd00a98f0 2024-03-25T19:42:21.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.427+0000 7f7db7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.427+0000 7f7db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.427+0000 7f7db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.427+0000 7f7db7fff640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:42:21.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.427+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dbc004eb0 con 0x7f7dd00a98f0 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2675424907 0 0) 0x7f7dd00abfa0 con 0x7f7dd013f710 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 >> v1:172.21.15.50:6790/0 conn(0x7f7dd01430c0 legacy=0x7f7dd01e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 >> v1:172.21.15.50:6789/0 conn(0x7f7dd00a98f0 legacy=0x7f7dd01387d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 1 monclient: found mon.b 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:21.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd0139010 con 0x7f7dd013f710 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 5 monclient: authenticate success, global_id 4276 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 10 monclient: _renew_subs 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd0137c50 con 0x7f7dd013f710 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 10 monclient: _renew_subs 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd01365c0 con 0x7f7dd013f710 2024-03-25T19:42:21.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.434+0000 7f7dd5daf640 1 librados: init done 2024-03-25T19:42:21.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.435+0000 7f7dd5daf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:21.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.435+0000 7f7dd5daf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:21.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.435+0000 7f7dd5daf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:21.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.435+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd01368c0 con 0x7f7dd013f710 2024-03-25T19:42:21.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.748+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dc4003610 con 0x7f7dd013f710 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.748+0000 7f7db7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.748+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7dc4001d80 con 0x7f7dd013f710 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.748+0000 7f7db7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.748+0000 7f7db7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.748+0000 7f7db7fff640 10 monclient: dump: 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:21.748 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:21.749 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:21.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.748+0000 7f7dcdffb640 4 set_mon_vals no callback set 2024-03-25T19:42:21.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.749+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f7dc4003ae0 con 0x7f7dd013f710 2024-03-25T19:42:21.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.750+0000 7f7dcf7fe640 10 monclient: get_auth_request con 0x7f7dbc06dd70 auth_method 0 2024-03-25T19:42:21.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.750+0000 7f7db7fff640 10 client.4276.objecter ms_handle_connect 0x7f7dbc06dd70 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f7dc4053590 con 0x7f7dd013f710 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 10 client.4276.objecter ms_dispatch 0x7f7dd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7dc404f0a0 con 0x7f7dd013f710 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:21.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.754+0000 7f7db7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:21.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.997+0000 7f7dd5daf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:21.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.997+0000 7f7dd5daf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:21.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.997+0000 7f7dd5daf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:21.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:21.997+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd0137de0 con 0x7f7dd013f710 2024-03-25T19:42:22.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.021+0000 7f7db7fff640 1 -- 172.21.15.50:0/1405051820 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f7dc4050fd0 con 0x7f7dd013f710 2024-03-25T19:42:22.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.021+0000 7f7db7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.021+0000 7f7db7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:22.020 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:42:22.021 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 10 librados: watch_flush enter 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 10 librados: watch_flush exit 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 10 client.4276.objecter successfully canceled tick 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 >> v1:172.21.15.76:6800/2572556811 conn(0x7f7dbc06dd70 legacy=0x7f7dbc070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 10 monclient: shutdown 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 >> v1:172.21.15.76:6789/0 conn(0x7f7dd013f710 legacy=0x7f7dd00ac3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 shutdown_connections 2024-03-25T19:42:22.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 >> 172.21.15.50:0/1405051820 conn(0x7f7dd00b7ea0 msgr2=0x7f7dd0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:22.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 shutdown_connections 2024-03-25T19:42:22.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 1 -- 172.21.15.50:0/1405051820 wait complete. 2024-03-25T19:42:22.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:22.023+0000 7f7dd5daf640 1 librados: shutdown 2024-03-25T19:42:22.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:28.841 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.950+0000 7f5a8c53e640 10 monclient: get_monmap_and_config 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.950+0000 7f5a8c53e640 10 monclient: build_initial_monmap 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.950+0000 7f5a8c53e640 10 monclient: monmap: 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:28.951558+0000 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:28.951558+0000 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:28.950 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:28.951 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:28.951 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:28.951 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:28.951 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:28.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.951+0000 7f5a8c53e640 1 Processor -- start 2024-03-25T19:42:28.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.951+0000 7f5a8c53e640 1 -- start start 2024-03-25T19:42:28.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.951+0000 7f5a8c53e640 10 monclient: init 2024-03-25T19:42:28.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:28.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:28.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8406c380 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:28.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 10 monclient(hunting): picked mon.noname-b con 0x7f5a8406d670 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:28.953 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 10 monclient(hunting): picked mon.noname-a con 0x7f5a840af8f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:28.953 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8406c9a0 con 0x7f5a840af8f0 2024-03-25T19:42:28.953 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840ada90 con 0x7f5a8406c380 2024-03-25T19:42:28.953 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840b0d10 con 0x7f5a8406d670 2024-03-25T19:42:28.953 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:28.953 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.952+0000 7f5a8c53e640 10 monclient(hunting): authenticate will time out at 4924.216095s 2024-03-25T19:42:28.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.957+0000 7f5a8a2da640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5a8406c380 0x7f5a840b0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:42868/0 (socket says 172.21.15.50:42868) 2024-03-25T19:42:28.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.957+0000 7f5a8a2da640 1 -- 172.21.15.50:0/4004611334 learned_addr learned my addr 172.21.15.50:0/4004611334 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:28.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.971+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752197925 0 0) 0x7f5a840ada90 con 0x7f5a8406c380 2024-03-25T19:42:28.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.971+0000 7f5a892d8640 10 monclient(hunting): _negotiate 2024-03-25T19:42:28.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.971+0000 7f5a892d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:28.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.971+0000 7f5a892d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:28.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.971+0000 7f5a892d8640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:42:28.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.971+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70003520 con 0x7f5a8406c380 2024-03-25T19:42:28.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216349001 0 0) 0x7f5a70003520 con 0x7f5a8406c380 2024-03-25T19:42:28.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 >> v1:172.21.15.50:6790/0 conn(0x7f5a8406d670 legacy=0x7f5a840ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 >> v1:172.21.15.50:6789/0 conn(0x7f5a840af8f0 legacy=0x7f5a840aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 10 monclient: _finish_hunting 0 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 1 monclient: found mon.noname-c 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a840add00 con 0x7f5a8406c380 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 10 monclient: _finish_auth 0 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 10 monclient: _check_auth_tickets 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5a80001690 con 0x7f5a8406c380 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a8c53e640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a8c53e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a892d8640 10 monclient: dump: 2024-03-25T19:42:28.984 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:28.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.983+0000 7f5a8c53e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a800017f0 con 0x7f5a8406c380 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a892d8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a892d8640 1 -- 172.21.15.50:0/4004611334 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5a80001c40 con 0x7f5a8406c380 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a892d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a892d8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a892d8640 10 monclient: dump: 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:28.996 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:28.997 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:28.997 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:28.997 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:28.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a8c53e640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:28.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.996+0000 7f5a8c53e640 4 set_mon_vals no callback set 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.997+0000 7f5a8c53e640 10 monclient: shutdown 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.997+0000 7f5a8c53e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.997+0000 7f5a8c53e640 1 -- 172.21.15.50:0/4004611334 >> v1:172.21.15.76:6789/0 conn(0x7f5a8406c380 legacy=0x7f5a840b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.997+0000 7f5a8c53e640 1 -- 172.21.15.50:0/4004611334 shutdown_connections 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.997+0000 7f5a8c53e640 1 -- 172.21.15.50:0/4004611334 >> 172.21.15.50:0/4004611334 conn(0x7f5a840b5c00 msgr2=0x7f5a840b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.997+0000 7f5a8c53e640 1 -- 172.21.15.50:0/4004611334 shutdown_connections 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.997+0000 7f5a8c53e640 1 -- 172.21.15.50:0/4004611334 wait complete. 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.998+0000 7f5a8c53e640 10 monclient: build_initial_monmap 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.998+0000 7f5a8c53e640 10 monclient: monmap: 2024-03-25T19:42:28.999 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.998+0000 7f5a8c53e640 1 librados: starting msgr at 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.998+0000 7f5a8c53e640 1 librados: starting objecter 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.998+0000 7f5a8c53e640 1 Processor -- start 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 1 -- start start 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 1 librados: setting wanted keys 2024-03-25T19:42:29.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 1 librados: calling monclient init 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient: init 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8406c380 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8406d670 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient(hunting): picked mon.noname-b con 0x7f5a840af8f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a841d3e00 con 0x7f5a8406c380 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a84143fe0 con 0x7f5a840af8f0 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a84144f00 con 0x7f5a8406d670 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:28.999+0000 7f5a8c53e640 10 monclient(hunting): authenticate will time out at 4924.263494s 2024-03-25T19:42:29.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.018+0000 7f5a8a2da640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5a8406c380 0x7f5a841438d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55660/0 (socket says 172.21.15.50:55660) 2024-03-25T19:42:29.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.018+0000 7f5a8a2da640 1 -- 172.21.15.50:0/436511853 learned_addr learned my addr 172.21.15.50:0/436511853 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:29.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.019+0000 7f5a7affd640 10 client.?.objecter ms_handle_connect 0x7f5a8406c380 2024-03-25T19:42:29.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.019+0000 7f5a7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.022+0000 7f5a7affd640 10 client.?.objecter ms_handle_connect 0x7f5a8406d670 2024-03-25T19:42:29.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.022+0000 7f5a7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.028+0000 7f5a7affd640 10 client.?.objecter ms_handle_connect 0x7f5a840af8f0 2024-03-25T19:42:29.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.028+0000 7f5a7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.034+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669370425 0 0) 0x7f5a841d3e00 con 0x7f5a8406c380 2024-03-25T19:42:29.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.034+0000 7f5a7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.034+0000 7f5a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.034+0000 7f5a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.034+0000 7f5a7affd640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:42:29.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.034+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a68004eb0 con 0x7f5a8406c380 2024-03-25T19:42:29.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.042+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156033133 0 0) 0x7f5a84144f00 con 0x7f5a8406d670 2024-03-25T19:42:29.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.042+0000 7f5a7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.042+0000 7f5a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.042+0000 7f5a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.042+0000 7f5a7affd640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:42:29.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.042+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a841d3e00 con 0x7f5a8406d670 2024-03-25T19:42:29.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.043+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000598506 0 0) 0x7f5a84143fe0 con 0x7f5a840af8f0 2024-03-25T19:42:29.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.043+0000 7f5a7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.043+0000 7f5a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.043+0000 7f5a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.043+0000 7f5a7affd640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:42:29.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.043+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a84144f00 con 0x7f5a840af8f0 2024-03-25T19:42:29.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.049+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 804251843 0 0) 0x7f5a68004eb0 con 0x7f5a8406c380 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.049+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a84143fe0 con 0x7f5a8406c380 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.049+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5a80001b00 con 0x7f5a8406c380 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.049+0000 7f5a7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.049+0000 7f5a7affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.049+0000 7f5a7affd640 10 monclient(hunting): dump: 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:29.049 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:29.050 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:29.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.049+0000 7f5a7affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:29.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.053+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 53935529 0 0) 0x7f5a841d3e00 con 0x7f5a8406d670 2024-03-25T19:42:29.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.053+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a68004eb0 con 0x7f5a8406d670 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.054+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5a74001930 con 0x7f5a8406d670 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.054+0000 7f5a7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.054+0000 7f5a7affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.054+0000 7f5a7affd640 10 monclient(hunting): dump: 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:29.054 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:29.055 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:29.055 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:29.055 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:29.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2523411996 0 0) 0x7f5a84143fe0 con 0x7f5a8406c380 2024-03-25T19:42:29.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 >> v1:172.21.15.50:6790/0 conn(0x7f5a8406d670 legacy=0x7f5a841d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 >> v1:172.21.15.76:6789/0 conn(0x7f5a840af8f0 legacy=0x7f5a841e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 1 monclient: found mon.a 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.060+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a841431f0 con 0x7f5a8406c380 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a7affd640 10 monclient: _finish_auth 0 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a7affd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 5 monclient: authenticate success, global_id 4299 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 10 monclient: _renew_subs 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8414eb30 con 0x7f5a8406c380 2024-03-25T19:42:29.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:29.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 10 monclient: _renew_subs 2024-03-25T19:42:29.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:29.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a841d7e80 con 0x7f5a8406c380 2024-03-25T19:42:29.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.061+0000 7f5a8c53e640 1 librados: init done 2024-03-25T19:42:29.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.062+0000 7f5a8c53e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.062+0000 7f5a8c53e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.062+0000 7f5a8c53e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:29.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.062+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a840627b0 con 0x7f5a8406c380 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.378+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a80001d40 con 0x7f5a8406c380 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.378+0000 7f5a7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.378+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5a80004d90 con 0x7f5a8406c380 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.378+0000 7f5a7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.378+0000 7f5a7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:29.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.378+0000 7f5a7affd640 10 monclient: dump: 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:29.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.378+0000 7f5a88ad7640 4 set_mon_vals no callback set 2024-03-25T19:42:29.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.379+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f5a8001ad60 con 0x7f5a8406c380 2024-03-25T19:42:29.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.380+0000 7f5a89ad9640 10 monclient: get_auth_request con 0x7f5a6806dea0 auth_method 0 2024-03-25T19:42:29.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.380+0000 7f5a7affd640 10 client.4299.objecter ms_handle_connect 0x7f5a6806dea0 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f5a800549a0 con 0x7f5a8406c380 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 10 client.4299.objecter ms_dispatch 0x7f5a84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a800502e0 con 0x7f5a8406c380 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.386+0000 7f5a7affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:29.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.644+0000 7f5a8c53e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.644+0000 7f5a8c53e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.644+0000 7f5a8c53e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:29.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.644+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a84069f40 con 0x7f5a8406c380 2024-03-25T19:42:29.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.685+0000 7f5a7affd640 1 -- 172.21.15.50:0/436511853 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f5a800523e0 con 0x7f5a8406c380 2024-03-25T19:42:29.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.685+0000 7f5a7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.685+0000 7f5a7affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:29.685 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:42:29.685 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.687+0000 7f5a8c53e640 10 librados: watch_flush enter 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 10 librados: watch_flush exit 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 10 client.4299.objecter successfully canceled tick 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 >> v1:172.21.15.76:6800/2572556811 conn(0x7f5a6806dea0 legacy=0x7f5a68070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 10 monclient: shutdown 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 >> v1:172.21.15.50:6789/0 conn(0x7f5a8406c380 legacy=0x7f5a841438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 shutdown_connections 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 >> 172.21.15.50:0/436511853 conn(0x7f5a840b5c00 msgr2=0x7f5a840b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 shutdown_connections 2024-03-25T19:42:29.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 1 -- 172.21.15.50:0/436511853 wait complete. 2024-03-25T19:42:29.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:29.688+0000 7f5a8c53e640 1 librados: shutdown 2024-03-25T19:42:29.700 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:34.097 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:54.603 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.105 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:54.611 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.113 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:54.619 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.130 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:01:54.636 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.164 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:54.635 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.170 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:54.641 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.176 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:54.682 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.203 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:01:54.674 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:34.216 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:01:54.722 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:36.506 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:36.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.626+0000 7f14853cb640 10 monclient: get_monmap_and_config 2024-03-25T19:42:36.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.626+0000 7f14853cb640 10 monclient: build_initial_monmap 2024-03-25T19:42:36.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.626+0000 7f14853cb640 10 monclient: monmap: 2024-03-25T19:42:36.626 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:36.627720+0000 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:36.627720+0000 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:36.627 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.627+0000 7f14853cb640 1 Processor -- start 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.627+0000 7f14853cb640 1 -- start start 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.627+0000 7f14853cb640 10 monclient: init 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.628+0000 7f14853cb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.628+0000 7f14853cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.628+0000 7f14853cb640 10 monclient(hunting): picked mon.noname-a con 0x7f14800a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.628+0000 7f14853cb640 10 monclient(hunting): picked mon.noname-c con 0x7f148013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:36.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.629+0000 7f14853cb640 10 monclient(hunting): picked mon.noname-b con 0x7f14801430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:36.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.629+0000 7f14853cb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1480142520 con 0x7f14800a98f0 2024-03-25T19:42:36.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.629+0000 7f14853cb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14800b28a0 con 0x7f148013f710 2024-03-25T19:42:36.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.629+0000 7f14853cb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14800b2a30 con 0x7f14801430c0 2024-03-25T19:42:36.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.629+0000 7f14853cb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:36.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.629+0000 7f14853cb640 10 monclient(hunting): authenticate will time out at 4931.892736s 2024-03-25T19:42:36.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.634+0000 7f1477fff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f148013f710 0x7f14801429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:47476/0 (socket says 172.21.15.50:47476) 2024-03-25T19:42:36.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.634+0000 7f1477fff640 1 -- 172.21.15.50:0/1293704657 learned_addr learned my addr 172.21.15.50:0/1293704657 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:36.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.650+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985388756 0 0) 0x7f14800b28a0 con 0x7f148013f710 2024-03-25T19:42:36.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.650+0000 7f147e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.650+0000 7f147e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.650+0000 7f147e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.650+0000 7f147e7fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:42:36.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.650+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f146c003520 con 0x7f148013f710 2024-03-25T19:42:36.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.660+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440148536 0 0) 0x7f1480142520 con 0x7f14800a98f0 2024-03-25T19:42:36.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.660+0000 7f147e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.660+0000 7f147e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.660+0000 7f147e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.660+0000 7f147e7fc640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:42:36.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.660+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14800b28a0 con 0x7f14800a98f0 2024-03-25T19:42:36.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685612889 0 0) 0x7f14800b2a30 con 0x7f14801430c0 2024-03-25T19:42:36.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1480142520 con 0x7f14801430c0 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095646537 0 0) 0x7f146c003520 con 0x7f148013f710 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 >> v1:172.21.15.50:6790/0 conn(0x7f14801430c0 legacy=0x7f148014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 >> v1:172.21.15.50:6789/0 conn(0x7f14800a98f0 legacy=0x7f14800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14800b44c0 con 0x7f148013f710 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:36.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1468001690 con 0x7f148013f710 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f147e7fc640 10 monclient: dump: 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.666 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:36.667 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:36.667 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:36.667 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:36.667 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:36.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f14853cb640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:42:36.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.664+0000 7f14853cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:36.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14680017f0 con 0x7f148013f710 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f147e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f147e7fc640 1 -- 172.21.15.50:0/1293704657 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1468001c00 con 0x7f148013f710 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f147e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f147e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f147e7fc640 10 monclient: dump: 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:36.688 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:36.689 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:36.689 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:36.689 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:36.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f14853cb640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:36.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.688+0000 7f14853cb640 4 set_mon_vals no callback set 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.689+0000 7f14853cb640 10 monclient: shutdown 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.689+0000 7f14853cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.689+0000 7f14853cb640 1 -- 172.21.15.50:0/1293704657 >> v1:172.21.15.76:6789/0 conn(0x7f148013f710 legacy=0x7f14801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.689+0000 7f14853cb640 1 -- 172.21.15.50:0/1293704657 shutdown_connections 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.689+0000 7f14853cb640 1 -- 172.21.15.50:0/1293704657 >> 172.21.15.50:0/1293704657 conn(0x7f14800b7ea0 msgr2=0x7f14800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.689+0000 7f14853cb640 1 -- 172.21.15.50:0/1293704657 shutdown_connections 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.689+0000 7f14853cb640 1 -- 172.21.15.50:0/1293704657 wait complete. 2024-03-25T19:42:36.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 10 monclient: build_initial_monmap 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 10 monclient: monmap: 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 1 librados: starting msgr at 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 1 librados: starting objecter 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 1 Processor -- start 2024-03-25T19:42:36.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 1 -- start start 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 1 librados: setting wanted keys 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 1 librados: calling monclient init 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.690+0000 7f14853cb640 10 monclient: init 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 10 monclient(hunting): picked mon.noname-c con 0x7f14800a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 10 monclient(hunting): picked mon.noname-a con 0x7f148013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 10 monclient(hunting): picked mon.noname-b con 0x7f14801430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1480138bf0 con 0x7f148013f710 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14801d7750 con 0x7f14801430c0 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14800abb00 con 0x7f14800a98f0 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:36.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.691+0000 7f14853cb640 10 monclient(hunting): authenticate will time out at 4931.955213s 2024-03-25T19:42:36.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.697+0000 7f1477fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f148013f710 0x7f14801d7be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:60052/0 (socket says 172.21.15.50:60052) 2024-03-25T19:42:36.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.697+0000 7f1477fff640 1 -- 172.21.15.50:0/1907959106 learned_addr learned my addr 172.21.15.50:0/1907959106 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:36.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.698+0000 7f14777fe640 10 client.?.objecter ms_handle_connect 0x7f148013f710 2024-03-25T19:42:36.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.698+0000 7f14777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:36.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.705+0000 7f14777fe640 10 client.?.objecter ms_handle_connect 0x7f14801430c0 2024-03-25T19:42:36.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.705+0000 7f14777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:36.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.706+0000 7f14777fe640 10 client.?.objecter ms_handle_connect 0x7f14800a98f0 2024-03-25T19:42:36.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.706+0000 7f14777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:36.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.709+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059915784 0 0) 0x7f1480138bf0 con 0x7f148013f710 2024-03-25T19:42:36.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.709+0000 7f14777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.709+0000 7f14777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.709+0000 7f14777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.709+0000 7f14777fe640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:42:36.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.709+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1460004eb0 con 0x7f148013f710 2024-03-25T19:42:36.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.719+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28826043 0 0) 0x7f14801d7750 con 0x7f14801430c0 2024-03-25T19:42:36.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.719+0000 7f14777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.719+0000 7f14777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.719+0000 7f14777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.719+0000 7f14777fe640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:42:36.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.719+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1480138bf0 con 0x7f14801430c0 2024-03-25T19:42:36.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.722+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530553088 0 0) 0x7f14800abb00 con 0x7f14800a98f0 2024-03-25T19:42:36.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.722+0000 7f14777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.722+0000 7f14777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.722+0000 7f14777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.722+0000 7f14777fe640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:42:36.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.722+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14801d7750 con 0x7f14800a98f0 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.738+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3269985937 0 0) 0x7f1460004eb0 con 0x7f148013f710 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.738+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14800abb00 con 0x7f148013f710 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.738+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1468001c00 con 0x7f148013f710 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.738+0000 7f14777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.738+0000 7f14777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.738+0000 7f14777fe640 10 monclient(hunting): dump: 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:36.738 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:36.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.738+0000 7f14777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.745+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 189382197 0 0) 0x7f1480138bf0 con 0x7f14801430c0 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.745+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1460004eb0 con 0x7f14801430c0 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.745+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1478001bd0 con 0x7f14801430c0 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.745+0000 7f14777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.745+0000 7f14777fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.745+0000 7f14777fe640 10 monclient(hunting): dump: 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.745 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:36.746 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:36.746 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:36.746 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:36.746 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:36.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.751+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 113161548 0 0) 0x7f14800abb00 con 0x7f148013f710 2024-03-25T19:42:36.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 >> v1:172.21.15.50:6790/0 conn(0x7f14800a98f0 legacy=0x7f14801384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 >> v1:172.21.15.76:6789/0 conn(0x7f14801430c0 legacy=0x7f1480133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:36.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 1 monclient: found mon.a 2024-03-25T19:42:36.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:36.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:36.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14801d41e0 con 0x7f148013f710 2024-03-25T19:42:36.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:36.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:36.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:36.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:42:36.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 10 monclient: _renew_subs 2024-03-25T19:42:36.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:36.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1480137a90 con 0x7f148013f710 2024-03-25T19:42:36.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 10 monclient: _renew_subs 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14801d82f0 con 0x7f148013f710 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.752+0000 7f14853cb640 1 librados: init done 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.754+0000 7f14853cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.754+0000 7f14853cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.754+0000 7f14853cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:36.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:36.754+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14800627b0 con 0x7f148013f710 2024-03-25T19:42:37.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.079+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14680011f0 con 0x7f148013f710 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.079+0000 7f14777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.079+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1468004cb0 con 0x7f148013f710 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.079+0000 7f14777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.079+0000 7f14777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.079+0000 7f14777fe640 10 monclient: dump: 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.079 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:37.080 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:37.080 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:37.080 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:37.080 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:37.080 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.079+0000 7f147dffb640 4 set_mon_vals no callback set 2024-03-25T19:42:37.080 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.080+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f146801acc0 con 0x7f148013f710 2024-03-25T19:42:37.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.080+0000 7f147effd640 10 monclient: get_auth_request con 0x7f146006dec0 auth_method 0 2024-03-25T19:42:37.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.081+0000 7f14777fe640 10 client.4305.objecter ms_handle_connect 0x7f146006dec0 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f14680549d0 con 0x7f148013f710 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 10 client.4305.objecter ms_dispatch 0x7f1480000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:37.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1468050310 con 0x7f148013f710 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.086+0000 7f14777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:37.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.303+0000 7f14853cb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.303+0000 7f14853cb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.303+0000 7f14853cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:42:37.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.303+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f148013dfd0 con 0x7f148013f710 2024-03-25T19:42:37.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.356+0000 7f14777fe640 1 -- 172.21.15.50:0/1907959106 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f1468052410 con 0x7f148013f710 2024-03-25T19:42:37.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.356+0000 7f14777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.356+0000 7f14777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:37.356 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:42:37.356 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:37.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 10 librados: watch_flush enter 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 10 librados: watch_flush exit 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 10 client.4305.objecter successfully canceled tick 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 >> v1:172.21.15.76:6800/2572556811 conn(0x7f146006dec0 legacy=0x7f1460070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 10 monclient: shutdown 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 >> v1:172.21.15.50:6789/0 conn(0x7f148013f710 legacy=0x7f14801d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 shutdown_connections 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 >> 172.21.15.50:0/1907959106 conn(0x7f14800b7ea0 msgr2=0x7f1480074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 shutdown_connections 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.358+0000 7f14853cb640 1 -- 172.21.15.50:0/1907959106 wait complete. 2024-03-25T19:42:37.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:37.359+0000 7f14853cb640 1 librados: shutdown 2024-03-25T19:42:37.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:44.175 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:44.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.290+0000 7fce328e6640 10 monclient: get_monmap_and_config 2024-03-25T19:42:44.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.291+0000 7fce328e6640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.291+0000 7fce328e6640 10 monclient: monmap: 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:44.291852+0000 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:44.291852+0000 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:44.291 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:44.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.291+0000 7fce328e6640 1 Processor -- start 2024-03-25T19:42:44.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.291+0000 7fce328e6640 1 -- start start 2024-03-25T19:42:44.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.291+0000 7fce328e6640 10 monclient: init 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c142520 con 0x7fce2c1430c0 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c0b28a0 con 0x7fce2c13f710 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c0b2a30 con 0x7fce2c0a98f0 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.292+0000 7fce328e6640 10 monclient(hunting): authenticate will time out at 4939.556532s 2024-03-25T19:42:44.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.298+0000 7fce2bfff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fce2c0a98f0 0x7fce2c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:36444/0 (socket says 172.21.15.50:36444) 2024-03-25T19:42:44.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.298+0000 7fce2bfff640 1 -- 172.21.15.50:0/2782338444 learned_addr learned my addr 172.21.15.50:0/2782338444 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:44.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.310+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093802345 0 0) 0x7fce2c0b2a30 con 0x7fce2c0a98f0 2024-03-25T19:42:44.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.310+0000 7fce2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.310+0000 7fce2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.310+0000 7fce2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.310+0000 7fce2affd640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:42:44.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.310+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce14003520 con 0x7fce2c0a98f0 2024-03-25T19:42:44.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.321+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080076230 0 0) 0x7fce2c142520 con 0x7fce2c1430c0 2024-03-25T19:42:44.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.321+0000 7fce2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.321+0000 7fce2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.321+0000 7fce2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.321+0000 7fce2affd640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:42:44.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.321+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce2c0b2a30 con 0x7fce2c1430c0 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2764067411 0 0) 0x7fce14003520 con 0x7fce2c0a98f0 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 >> v1:172.21.15.76:6789/0 conn(0x7fce2c13f710 legacy=0x7fce2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 >> v1:172.21.15.50:6789/0 conn(0x7fce2c1430c0 legacy=0x7fce2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 1 monclient: found mon.noname-b 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c0b44c0 con 0x7fce2c0a98f0 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fce1c001690 con 0x7fce2c0a98f0 2024-03-25T19:42:44.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce2affd640 10 monclient: dump: 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce328e6640 5 monclient: authenticate success, global_id 4292 2024-03-25T19:42:44.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.322+0000 7fce328e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce1c0017f0 con 0x7fce2c0a98f0 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce2affd640 1 -- 172.21.15.50:0/2782338444 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fce1c001c00 con 0x7fce2c0a98f0 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce328e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce328e6640 4 set_mon_vals no callback set 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.325+0000 7fce2affd640 10 monclient: dump: 2024-03-25T19:42:44.325 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:44.326 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:44.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.326+0000 7fce328e6640 10 monclient: shutdown 2024-03-25T19:42:44.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.326+0000 7fce328e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.326+0000 7fce328e6640 1 -- 172.21.15.50:0/2782338444 >> v1:172.21.15.50:6790/0 conn(0x7fce2c0a98f0 legacy=0x7fce2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.326+0000 7fce328e6640 1 -- 172.21.15.50:0/2782338444 shutdown_connections 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.326+0000 7fce328e6640 1 -- 172.21.15.50:0/2782338444 >> 172.21.15.50:0/2782338444 conn(0x7fce2c0b7ea0 msgr2=0x7fce2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.326+0000 7fce328e6640 1 -- 172.21.15.50:0/2782338444 shutdown_connections 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.326+0000 7fce328e6640 1 -- 172.21.15.50:0/2782338444 wait complete. 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.327+0000 7fce328e6640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.327+0000 7fce328e6640 10 monclient: monmap: 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:44.329 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.327+0000 7fce328e6640 1 librados: starting msgr at 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.327+0000 7fce328e6640 1 librados: starting objecter 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 1 Processor -- start 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 1 -- start start 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 1 librados: setting wanted keys 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 1 librados: calling monclient init 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 10 monclient: init 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.328+0000 7fce328e6640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.329+0000 7fce328e6640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:44.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.329+0000 7fce328e6640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:44.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.329+0000 7fce328e6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c13be70 con 0x7fce2c1430c0 2024-03-25T19:42:44.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.329+0000 7fce328e6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c131660 con 0x7fce2c13f710 2024-03-25T19:42:44.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.329+0000 7fce328e6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c1317f0 con 0x7fce2c0a98f0 2024-03-25T19:42:44.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.329+0000 7fce328e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.329+0000 7fce328e6640 10 monclient(hunting): authenticate will time out at 4939.592854s 2024-03-25T19:42:44.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.337+0000 7fce30e83640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fce2c1430c0 0x7fce2c1e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:39138/0 (socket says 172.21.15.50:39138) 2024-03-25T19:42:44.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.337+0000 7fce30e83640 1 -- 172.21.15.50:0/1063248977 learned_addr learned my addr 172.21.15.50:0/1063248977 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:44.344 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.345+0000 7fce28ff9640 10 client.?.objecter ms_handle_connect 0x7fce2c1430c0 2024-03-25T19:42:44.344 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.345+0000 7fce28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.350+0000 7fce28ff9640 10 client.?.objecter ms_handle_connect 0x7fce2c0a98f0 2024-03-25T19:42:44.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.350+0000 7fce28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.358+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123152788 0 0) 0x7fce2c13be70 con 0x7fce2c1430c0 2024-03-25T19:42:44.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.358+0000 7fce28ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.358+0000 7fce28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.358+0000 7fce28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.358+0000 7fce28ff9640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:42:44.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.358+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18004eb0 con 0x7fce2c1430c0 2024-03-25T19:42:44.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.362+0000 7fce28ff9640 10 client.?.objecter ms_handle_connect 0x7fce2c13f710 2024-03-25T19:42:44.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.362+0000 7fce28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.363+0000 7fce30e83640 1 -- 172.21.15.50:0/1063248977 >> v1:172.21.15.50:6789/0 conn(0x7fce2c1430c0 legacy=0x7fce2c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:42:44.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.363+0000 7fce30e83640 1 -- 172.21.15.50:0/1063248977 >> v1:172.21.15.50:6789/0 conn(0x7fce2c1430c0 legacy=0x7fce2c1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:42:44.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.363+0000 7fce30e83640 1 --1- 172.21.15.50:0/1063248977 >> v1:172.21.15.50:6789/0 conn(0x7fce2c1430c0 0x7fce2c1e2fc0 :-1 s=OPENED pgs=8311 cs=1 l=1).handle_message read tag failed 2024-03-25T19:42:44.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.363+0000 7fce30e83640 1 --1- 172.21.15.50:0/1063248977 >> v1:172.21.15.50:6789/0 conn(0x7fce2c1430c0 0x7fce2c1e2fc0 :-1 s=OPENED pgs=8311 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:42:44.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.363+0000 7fce28ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.50:6789/0 2024-03-25T19:42:44.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.372+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321518846 0 0) 0x7fce2c1317f0 con 0x7fce2c0a98f0 2024-03-25T19:42:44.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.372+0000 7fce28ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.372+0000 7fce28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.372+0000 7fce28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.372+0000 7fce28ff9640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:42:44.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.372+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce2c13be70 con 0x7fce2c0a98f0 2024-03-25T19:42:44.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.375+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439495441 0 0) 0x7fce2c131660 con 0x7fce2c13f710 2024-03-25T19:42:44.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.375+0000 7fce28ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.375+0000 7fce28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.375+0000 7fce28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.375+0000 7fce28ff9640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:42:44.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.375+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce2c1317f0 con 0x7fce2c13f710 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.390+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1450402143 0 0) 0x7fce2c1317f0 con 0x7fce2c13f710 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.390+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce2c131660 con 0x7fce2c13f710 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.390+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fce20001860 con 0x7fce2c13f710 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.390+0000 7fce28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.390+0000 7fce28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.390+0000 7fce28ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:44.390 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:44.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.390+0000 7fce28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:44.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.399+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3537800366 0 0) 0x7fce2c13be70 con 0x7fce2c0a98f0 2024-03-25T19:42:44.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.399+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce2c1317f0 con 0x7fce2c0a98f0 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.399+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fce1c001b00 con 0x7fce2c0a98f0 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.399+0000 7fce28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.399+0000 7fce28ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.399+0000 7fce28ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:44.399 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:44.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2545798189 0 0) 0x7fce2c131660 con 0x7fce2c13f710 2024-03-25T19:42:44.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 >> v1:172.21.15.50:6790/0 conn(0x7fce2c0a98f0 legacy=0x7fce2c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 >> v1:172.21.15.50:6789/0 conn(0x7fce2c1430c0 legacy=0x7fce2c1e2fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:42:44.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 1 monclient: found mon.b 2024-03-25T19:42:44.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c131980 con 0x7fce2c13f710 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce328e6640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce328e6640 10 monclient: _renew_subs 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce328e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.411+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2c135c60 con 0x7fce2c13f710 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 10 monclient: _renew_subs 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2c0b7770 con 0x7fce2c13f710 2024-03-25T19:42:44.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 1 librados: init done 2024-03-25T19:42:44.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:44.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.412+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c1371d0 con 0x7fce2c13f710 2024-03-25T19:42:44.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.690+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce20001d30 con 0x7fce2c13f710 2024-03-25T19:42:44.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.690+0000 7fce28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:44.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.690+0000 7fce2a7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.691+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fce20003790 con 0x7fce2c13f710 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.691+0000 7fce28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.691+0000 7fce28ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.691+0000 7fce28ff9640 10 monclient: dump: 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:44.691 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:44.692 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:44.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.691+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fce20019760 con 0x7fce2c13f710 2024-03-25T19:42:44.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.692+0000 7fce2bfff640 10 monclient: get_auth_request con 0x7fce1806dfc0 auth_method 0 2024-03-25T19:42:44.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.692+0000 7fce28ff9640 10 client.4294.objecter ms_handle_connect 0x7fce1806dfc0 2024-03-25T19:42:44.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fce20053440 con 0x7fce2c13f710 2024-03-25T19:42:44.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 10 client.4294.objecter ms_dispatch 0x7fce2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:44.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:44.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:44.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T19:42:44.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce2004ef50 con 0x7fce2c13f710 2024-03-25T19:42:44.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.696+0000 7fce28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:44.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.926+0000 7fce328e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:44.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.926+0000 7fce328e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:44.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.926+0000 7fce328e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:44.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.926+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce2c074cc0 con 0x7fce2c13f710 2024-03-25T19:42:44.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.985+0000 7fce28ff9640 1 -- 172.21.15.50:0/1063248977 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fce20050e80 con 0x7fce2c13f710 2024-03-25T19:42:44.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.985+0000 7fce28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:44.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.985+0000 7fce28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:44.985 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:42:44.985 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:44.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 10 librados: watch_flush enter 2024-03-25T19:42:44.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 10 librados: watch_flush exit 2024-03-25T19:42:44.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T19:42:44.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 10 client.4294.objecter successfully canceled tick 2024-03-25T19:42:44.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 >> v1:172.21.15.76:6800/2572556811 conn(0x7fce1806dfc0 legacy=0x7fce18070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 10 monclient: shutdown 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 >> v1:172.21.15.76:6789/0 conn(0x7fce2c13f710 legacy=0x7fce2c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 shutdown_connections 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 >> 172.21.15.50:0/1063248977 conn(0x7fce2c0b7ea0 msgr2=0x7fce2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 shutdown_connections 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 1 -- 172.21.15.50:0/1063248977 wait complete. 2024-03-25T19:42:44.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:44.987+0000 7fce328e6640 1 librados: shutdown 2024-03-25T19:42:45.000 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:51.807 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:51.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.926+0000 7f671b10e640 10 monclient: get_monmap_and_config 2024-03-25T19:42:51.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.926+0000 7f671b10e640 10 monclient: build_initial_monmap 2024-03-25T19:42:51.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.926+0000 7f671b10e640 10 monclient: monmap: 2024-03-25T19:42:51.927 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:51.927 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:51.927 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:51.928592+0000 2024-03-25T19:42:51.927 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:51.928592+0000 2024-03-25T19:42:51.928 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:51.928 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:51.928 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:51.928 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:51.928 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:51.928 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:51.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.927+0000 7f671b10e640 1 Processor -- start 2024-03-25T19:42:51.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.927+0000 7f671b10e640 1 -- start start 2024-03-25T19:42:51.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.927+0000 7f671b10e640 10 monclient: init 2024-03-25T19:42:51.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:51.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:51.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 10 monclient(hunting): picked mon.noname-a con 0x7f67140a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:51.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 10 monclient(hunting): picked mon.noname-c con 0x7f671413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 10 monclient(hunting): picked mon.noname-b con 0x7f67141430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6714142520 con 0x7f67140a98f0 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67140b28a0 con 0x7f671413f710 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67140b2a30 con 0x7f67141430c0 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.928+0000 7f671b10e640 10 monclient(hunting): authenticate will time out at 4947.193189s 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.930+0000 7f670bfff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f671413f710 0x7f67141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:50462/0 (socket says 172.21.15.50:50462) 2024-03-25T19:42:51.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:51.930+0000 7f670bfff640 1 -- 172.21.15.50:0/3679307060 learned_addr learned my addr 172.21.15.50:0/3679307060 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:52.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.025+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202476166 0 0) 0x7f67140b2a30 con 0x7f67141430c0 2024-03-25T19:42:52.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.025+0000 7f670b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.025+0000 7f670b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.025+0000 7f670b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.025+0000 7f670b7fe640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:42:52.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.025+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66fc003520 con 0x7f67141430c0 2024-03-25T19:42:52.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.026+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397089884 0 0) 0x7f6714142520 con 0x7f67140a98f0 2024-03-25T19:42:52.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.026+0000 7f670b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.026+0000 7f670b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.026+0000 7f670b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.026+0000 7f670b7fe640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:42:52.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.026+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67140b2a30 con 0x7f67140a98f0 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729452900 0 0) 0x7f67140b2a30 con 0x7f67140a98f0 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 >> v1:172.21.15.50:6790/0 conn(0x7f67141430c0 legacy=0x7f671414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 >> v1:172.21.15.76:6789/0 conn(0x7f671413f710 legacy=0x7f67141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67140b44c0 con 0x7f67140a98f0 2024-03-25T19:42:52.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6710001670 con 0x7f67140a98f0 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f670b7fe640 10 monclient: dump: 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.051 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:52.052 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:52.052 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:52.052 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:52.052 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:52.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f671b10e640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:42:52.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.049+0000 7f671b10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67100017d0 con 0x7f67140a98f0 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f670b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f670b7fe640 1 -- 172.21.15.50:0/3679307060 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6710001ba0 con 0x7f67140a98f0 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f670b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f670b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f670b7fe640 10 monclient: dump: 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.057 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f671b10e640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:52.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.056+0000 7f671b10e640 4 set_mon_vals no callback set 2024-03-25T19:42:52.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.057+0000 7f671b10e640 10 monclient: shutdown 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.057+0000 7f671b10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.057+0000 7f671b10e640 1 -- 172.21.15.50:0/3679307060 >> v1:172.21.15.50:6789/0 conn(0x7f67140a98f0 legacy=0x7f67140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.057+0000 7f671b10e640 1 -- 172.21.15.50:0/3679307060 shutdown_connections 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.057+0000 7f671b10e640 1 -- 172.21.15.50:0/3679307060 >> 172.21.15.50:0/3679307060 conn(0x7f67140b7ea0 msgr2=0x7f67140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.057+0000 7f671b10e640 1 -- 172.21.15.50:0/3679307060 shutdown_connections 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.057+0000 7f671b10e640 1 -- 172.21.15.50:0/3679307060 wait complete. 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.058+0000 7f671b10e640 10 monclient: build_initial_monmap 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.058+0000 7f671b10e640 10 monclient: monmap: 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:52.060 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.058+0000 7f671b10e640 1 librados: starting msgr at 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.058+0000 7f671b10e640 1 librados: starting objecter 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.059+0000 7f671b10e640 1 Processor -- start 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.059+0000 7f671b10e640 1 -- start start 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.059+0000 7f671b10e640 1 librados: setting wanted keys 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.059+0000 7f671b10e640 1 librados: calling monclient init 2024-03-25T19:42:52.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.059+0000 7f671b10e640 10 monclient: init 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 10 monclient(hunting): picked mon.noname-c con 0x7f67140a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 10 monclient(hunting): picked mon.noname-b con 0x7f671413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 10 monclient(hunting): picked mon.noname-a con 0x7f67141430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6714138630 con 0x7f67141430c0 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6714139280 con 0x7f671413f710 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67141349e0 con 0x7f67140a98f0 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:52.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.060+0000 7f671b10e640 10 monclient(hunting): authenticate will time out at 4947.325108s 2024-03-25T19:42:52.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.066+0000 7f670bfff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f671413f710 0x7f6714133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:50504/0 (socket says 172.21.15.50:50504) 2024-03-25T19:42:52.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.066+0000 7f670bfff640 1 -- 172.21.15.50:0/2703562697 learned_addr learned my addr 172.21.15.50:0/2703562697 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:52.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.067+0000 7f67097fa640 10 client.?.objecter ms_handle_connect 0x7f671413f710 2024-03-25T19:42:52.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.067+0000 7f67097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.073+0000 7f67097fa640 10 client.?.objecter ms_handle_connect 0x7f67141430c0 2024-03-25T19:42:52.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.073+0000 7f67097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.077+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516022480 0 0) 0x7f6714139280 con 0x7f671413f710 2024-03-25T19:42:52.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.077+0000 7f67097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.077+0000 7f67097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.077+0000 7f67097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.077+0000 7f67097fa640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:42:52.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.077+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6700004eb0 con 0x7f671413f710 2024-03-25T19:42:52.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.078+0000 7f67097fa640 10 client.?.objecter ms_handle_connect 0x7f67140a98f0 2024-03-25T19:42:52.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.078+0000 7f67097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.081+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221443782 0 0) 0x7f6714138630 con 0x7f67141430c0 2024-03-25T19:42:52.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.081+0000 7f67097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.081+0000 7f67097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.081+0000 7f67097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.081+0000 7f67097fa640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:42:52.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.081+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6714139280 con 0x7f67141430c0 2024-03-25T19:42:52.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.089+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 600181637 0 0) 0x7f6700004eb0 con 0x7f671413f710 2024-03-25T19:42:52.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.089+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6714138630 con 0x7f671413f710 2024-03-25T19:42:52.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.089+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f67040018b0 con 0x7f671413f710 2024-03-25T19:42:52.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.089+0000 7f67097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.089+0000 7f67097fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:52.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.089+0000 7f67097fa640 10 monclient(hunting): dump: 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.089+0000 7f67097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.090+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284669535 0 0) 0x7f67141349e0 con 0x7f67140a98f0 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.090+0000 7f67097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.090+0000 7f67097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.090+0000 7f67097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.090+0000 7f67097fa640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:42:52.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.090+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6700004eb0 con 0x7f67140a98f0 2024-03-25T19:42:52.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.110+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 429901871 0 0) 0x7f6714139280 con 0x7f67141430c0 2024-03-25T19:42:52.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.111+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67141349e0 con 0x7f67141430c0 2024-03-25T19:42:52.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.111+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f670c001bd0 con 0x7f67141430c0 2024-03-25T19:42:52.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.111+0000 7f67097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.111+0000 7f67097fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.111+0000 7f67097fa640 10 monclient(hunting): dump: 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:52.112 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3755671037 0 0) 0x7f6714138630 con 0x7f671413f710 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 >> v1:172.21.15.50:6790/0 conn(0x7f67140a98f0 legacy=0x7f67141d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 >> v1:172.21.15.50:6789/0 conn(0x7f67141430c0 legacy=0x7f67141342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 1 monclient: found mon.b 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714139560 con 0x7f671413f710 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f67097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 10 monclient: _renew_subs 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6714139840 con 0x7f671413f710 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 10 monclient: _renew_subs 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67141381d0 con 0x7f671413f710 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.112+0000 7f671b10e640 1 librados: init done 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.113+0000 7f671b10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.113+0000 7f671b10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.113+0000 7f671b10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:52.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.113+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67140627b0 con 0x7f671413f710 2024-03-25T19:42:52.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.399+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67040047e0 con 0x7f671413f710 2024-03-25T19:42:52.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.399+0000 7f67097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.399+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6704004c30 con 0x7f671413f710 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.399+0000 7f67097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.399+0000 7f67097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.399+0000 7f67097fa640 10 monclient: dump: 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:52.401 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:52.402 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:52.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.399+0000 7f670affd640 4 set_mon_vals no callback set 2024-03-25T19:42:52.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.400+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f6704004e50 con 0x7f671413f710 2024-03-25T19:42:52.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.401+0000 7f6718eaa640 10 monclient: get_auth_request con 0x7f670006dfd0 auth_method 0 2024-03-25T19:42:52.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.401+0000 7f67097fa640 10 client.4300.objecter ms_handle_connect 0x7f670006dfd0 2024-03-25T19:42:52.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.405+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f67040547c0 con 0x7f671413f710 2024-03-25T19:42:52.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.405+0000 7f67097fa640 10 client.4300.objecter ms_dispatch 0x7f6714000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:52.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.405+0000 7f67097fa640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:52.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.406+0000 7f67097fa640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:52.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.406+0000 7f67097fa640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T19:42:52.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.406+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67040502d0 con 0x7f671413f710 2024-03-25T19:42:52.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.406+0000 7f67097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.406+0000 7f67097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:52.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.652+0000 7f671b10e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.652+0000 7f671b10e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.652+0000 7f671b10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:52.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.652+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67141387e0 con 0x7f671413f710 2024-03-25T19:42:52.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.679+0000 7f67097fa640 1 -- 172.21.15.50:0/2703562697 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f67141387e0 con 0x7f671413f710 2024-03-25T19:42:52.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.679+0000 7f67097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.679+0000 7f67097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:52.679 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:42:52.680 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 10 librados: watch_flush enter 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 10 librados: watch_flush exit 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 10 client.4300.objecter successfully canceled tick 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 >> v1:172.21.15.76:6800/2572556811 conn(0x7f670006dfd0 legacy=0x7f6700070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 10 monclient: shutdown 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 >> v1:172.21.15.76:6789/0 conn(0x7f671413f710 legacy=0x7f6714133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 shutdown_connections 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 >> 172.21.15.50:0/2703562697 conn(0x7f67140b7ea0 msgr2=0x7f6714074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 shutdown_connections 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 1 -- 172.21.15.50:0/2703562697 wait complete. 2024-03-25T19:42:52.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:52.681+0000 7f671b10e640 1 librados: shutdown 2024-03-25T19:42:52.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:54.205 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:14.711 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.218 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:14.724 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.247 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:14.753 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.273 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:14.779 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.278 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:14.784 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.286 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:14.757 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.289 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:14.759 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.318 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:14.824 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:54.333 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:14.804 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.499 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:59.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.617+0000 7f594edd0640 10 monclient: get_monmap_and_config 2024-03-25T19:42:59.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.617+0000 7f594edd0640 10 monclient: build_initial_monmap 2024-03-25T19:42:59.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.617+0000 7f594edd0640 10 monclient: monmap: 2024-03-25T19:42:59.617 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:59.617 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:59.617 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:42:59.618543+0000 2024-03-25T19:42:59.617 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:42:59.618543+0000 2024-03-25T19:42:59.618 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:59.618 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:59.618 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:59.618 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:42:59.618 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:42:59.618 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.618+0000 7f594edd0640 1 Processor -- start 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.618+0000 7f594edd0640 1 -- start start 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.618+0000 7f594edd0640 10 monclient: init 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 10 monclient(hunting): picked mon.noname-b con 0x7f59480a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 10 monclient(hunting): picked mon.noname-c con 0x7f594813f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 10 monclient(hunting): picked mon.noname-a con 0x7f59481430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:59.619 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5948142520 con 0x7f59481430c0 2024-03-25T19:42:59.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59480b28a0 con 0x7f594813f710 2024-03-25T19:42:59.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59480b2a30 con 0x7f59480a98f0 2024-03-25T19:42:59.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:59.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.619+0000 7f594edd0640 10 monclient(hunting): authenticate will time out at 4954.883212s 2024-03-25T19:42:59.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.635+0000 7f594d36d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f59481430c0 0x7f594814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44750/0 (socket says 172.21.15.50:44750) 2024-03-25T19:42:59.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.635+0000 7f594d36d640 1 -- 172.21.15.50:0/137175170 learned_addr learned my addr 172.21.15.50:0/137175170 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:59.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.648+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171157794 0 0) 0x7f5948142520 con 0x7f59481430c0 2024-03-25T19:42:59.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.648+0000 7f59477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.648+0000 7f59477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.648+0000 7f59477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.648+0000 7f59477fe640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:42:59.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.648+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5938003520 con 0x7f59481430c0 2024-03-25T19:42:59.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.650+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624626712 0 0) 0x7f59480b28a0 con 0x7f594813f710 2024-03-25T19:42:59.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.650+0000 7f59477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.650+0000 7f59477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.650+0000 7f59477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.650+0000 7f59477fe640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:42:59.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.650+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5948142520 con 0x7f594813f710 2024-03-25T19:42:59.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.657+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337747264 0 0) 0x7f59480b2a30 con 0x7f59480a98f0 2024-03-25T19:42:59.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.657+0000 7f59477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.657+0000 7f59477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.657+0000 7f59477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.657+0000 7f59477fe640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:42:59.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.657+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59480b28a0 con 0x7f59480a98f0 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2321325313 0 0) 0x7f59480b28a0 con 0x7f59480a98f0 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 >> v1:172.21.15.76:6789/0 conn(0x7f594813f710 legacy=0x7f59481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 >> v1:172.21.15.50:6789/0 conn(0x7f59481430c0 legacy=0x7f594814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 1 monclient: found mon.noname-b 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59480b44c0 con 0x7f59480a98f0 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f593c001930 con 0x7f59480a98f0 2024-03-25T19:42:59.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f59477fe640 10 monclient: dump: 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f594edd0640 5 monclient: authenticate success, global_id 4304 2024-03-25T19:42:59.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.671+0000 7f594edd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f593c001a90 con 0x7f59480a98f0 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f59477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f59477fe640 1 -- 172.21.15.50:0/137175170 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f593c001ee0 con 0x7f59480a98f0 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f59477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f59477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f59477fe640 10 monclient: dump: 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:59.681 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:59.682 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:59.682 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:59.682 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:59.682 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:59.682 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:59.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f594edd0640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:59.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.681+0000 7f594edd0640 4 set_mon_vals no callback set 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.682+0000 7f594edd0640 10 monclient: shutdown 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.682+0000 7f594edd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.682+0000 7f594edd0640 1 -- 172.21.15.50:0/137175170 >> v1:172.21.15.50:6790/0 conn(0x7f59480a98f0 legacy=0x7f59480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.682+0000 7f594edd0640 1 -- 172.21.15.50:0/137175170 shutdown_connections 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.682+0000 7f594edd0640 1 -- 172.21.15.50:0/137175170 >> 172.21.15.50:0/137175170 conn(0x7f59480b7ea0 msgr2=0x7f59480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.682+0000 7f594edd0640 1 -- 172.21.15.50:0/137175170 shutdown_connections 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.682+0000 7f594edd0640 1 -- 172.21.15.50:0/137175170 wait complete. 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.683+0000 7f594edd0640 10 monclient: build_initial_monmap 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.683+0000 7f594edd0640 10 monclient: monmap: 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:42:59.684 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.684+0000 7f594edd0640 1 librados: starting msgr at 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.684+0000 7f594edd0640 1 librados: starting objecter 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.684+0000 7f594edd0640 1 Processor -- start 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.684+0000 7f594edd0640 1 -- start start 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.684+0000 7f594edd0640 1 librados: setting wanted keys 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.684+0000 7f594edd0640 1 librados: calling monclient init 2024-03-25T19:42:59.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.684+0000 7f594edd0640 10 monclient: init 2024-03-25T19:42:59.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:59.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:59.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 10 monclient(hunting): picked mon.noname-c con 0x7f59480a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:42:59.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 10 monclient(hunting): picked mon.noname-a con 0x7f594813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:42:59.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 10 monclient(hunting): picked mon.noname-b con 0x7f59481430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:42:59.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59481d78a0 con 0x7f594813f710 2024-03-25T19:42:59.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59481d51f0 con 0x7f59481430c0 2024-03-25T19:42:59.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59481d8440 con 0x7f59480a98f0 2024-03-25T19:42:59.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:59.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.685+0000 7f594edd0640 10 monclient(hunting): authenticate will time out at 4954.949008s 2024-03-25T19:42:59.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.689+0000 7f5947fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f594813f710 0x7f59481385a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:44798/0 (socket says 172.21.15.50:44798) 2024-03-25T19:42:59.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.689+0000 7f5947fff640 1 -- 172.21.15.50:0/356141383 learned_addr learned my addr 172.21.15.50:0/356141383 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:42:59.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.692+0000 7f59457fa640 10 client.?.objecter ms_handle_connect 0x7f594813f710 2024-03-25T19:42:59.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.693+0000 7f59457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:59.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.703+0000 7f59457fa640 10 client.?.objecter ms_handle_connect 0x7f59481430c0 2024-03-25T19:42:59.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.703+0000 7f59457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:59.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.705+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804645833 0 0) 0x7f59481d78a0 con 0x7f594813f710 2024-03-25T19:42:59.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.705+0000 7f59457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.705+0000 7f59457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.705+0000 7f59457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.705+0000 7f59457fa640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:42:59.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.705+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f592c004eb0 con 0x7f594813f710 2024-03-25T19:42:59.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.708+0000 7f59457fa640 10 client.?.objecter ms_handle_connect 0x7f59480a98f0 2024-03-25T19:42:59.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.708+0000 7f59457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:59.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.714+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517782764 0 0) 0x7f59481d51f0 con 0x7f59481430c0 2024-03-25T19:42:59.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.714+0000 7f59457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.714+0000 7f59457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.714+0000 7f59457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.714+0000 7f59457fa640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:42:59.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.714+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59481d78a0 con 0x7f59481430c0 2024-03-25T19:42:59.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.725+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953380716 0 0) 0x7f59481d8440 con 0x7f59480a98f0 2024-03-25T19:42:59.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.725+0000 7f59457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.725+0000 7f59457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.725+0000 7f59457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.725+0000 7f59457fa640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:42:59.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.725+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59481d51f0 con 0x7f59480a98f0 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.729+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1521463955 0 0) 0x7f59481d78a0 con 0x7f59481430c0 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.729+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59481d8440 con 0x7f59481430c0 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.729+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5940001730 con 0x7f59481430c0 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.729+0000 7f59457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.729+0000 7f59457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.729+0000 7f59457fa640 10 monclient(hunting): dump: 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:59.729 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:42:59.730 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:59.730 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:42:59.730 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:42:59.730 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:42:59.730 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:42:59.730 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:42:59.730 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.729+0000 7f59457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:59.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2061734311 0 0) 0x7f59481d8440 con 0x7f59481430c0 2024-03-25T19:42:59.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 >> v1:172.21.15.50:6790/0 conn(0x7f59480a98f0 legacy=0x7f59481d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 >> v1:172.21.15.50:6789/0 conn(0x7f594813f710 legacy=0x7f59481385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 1 monclient: found mon.b 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5948138cb0 con 0x7f59481430c0 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f59457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f594edd0640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f594edd0640 10 monclient: _renew_subs 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f594edd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:59.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.736+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5948137890 con 0x7f59481430c0 2024-03-25T19:42:59.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:59.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 10 monclient: _renew_subs 2024-03-25T19:42:59.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:59.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5948137c30 con 0x7f59481430c0 2024-03-25T19:42:59.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 1 librados: init done 2024-03-25T19:42:59.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:59.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:59.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:42:59.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:42:59.737+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59481d7600 con 0x7f59481430c0 2024-03-25T19:43:00.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.017+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5940001c20 con 0x7f59481430c0 2024-03-25T19:43:00.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.017+0000 7f59457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:00.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.017+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5940004980 con 0x7f59481430c0 2024-03-25T19:43:00.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.017+0000 7f59457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.017+0000 7f59457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.017+0000 7f59457fa640 10 monclient: dump: 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:00.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.017+0000 7f5946ffd640 4 set_mon_vals no callback set 2024-03-25T19:43:00.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.018+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f594001a950 con 0x7f59481430c0 2024-03-25T19:43:00.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.019+0000 7f594cb6c640 10 monclient: get_auth_request con 0x7f592c06ddd0 auth_method 0 2024-03-25T19:43:00.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.019+0000 7f59457fa640 10 client.4306.objecter ms_handle_connect 0x7f592c06ddd0 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f5940054690 con 0x7f59481430c0 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 10 client.4306.objecter ms_dispatch 0x7f5948000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f594004ffd0 con 0x7f59481430c0 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.023+0000 7f59457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:00.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.279+0000 7f594edd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.279+0000 7f594edd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.279+0000 7f594edd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:00.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.279+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59480627b0 con 0x7f59481430c0 2024-03-25T19:43:00.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.301+0000 7f59457fa640 1 -- 172.21.15.50:0/356141383 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f59400520d0 con 0x7f59481430c0 2024-03-25T19:43:00.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.301+0000 7f59457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.301+0000 7f59457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:00.301 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:00.301 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:00.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.303+0000 7f594edd0640 10 librados: watch_flush enter 2024-03-25T19:43:00.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.303+0000 7f594edd0640 10 librados: watch_flush exit 2024-03-25T19:43:00.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.303+0000 7f594edd0640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T19:43:00.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.303+0000 7f594edd0640 10 client.4306.objecter successfully canceled tick 2024-03-25T19:43:00.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 >> v1:172.21.15.76:6800/2572556811 conn(0x7f592c06ddd0 legacy=0x7f592c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 10 monclient: shutdown 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 >> v1:172.21.15.76:6789/0 conn(0x7f59481430c0 legacy=0x7f59481e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 shutdown_connections 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 >> 172.21.15.50:0/356141383 conn(0x7f59480b7ea0 msgr2=0x7f5948074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 shutdown_connections 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 1 -- 172.21.15.50:0/356141383 wait complete. 2024-03-25T19:43:00.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:00.304+0000 7f594edd0640 1 librados: shutdown 2024-03-25T19:43:00.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:07.124 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:07.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.228+0000 7fe88e4e2640 10 monclient: get_monmap_and_config 2024-03-25T19:43:07.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.228+0000 7fe88e4e2640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.228+0000 7fe88e4e2640 10 monclient: monmap: 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:43:07.229051+0000 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:43:07.229051+0000 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:43:07.228 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.229+0000 7fe88e4e2640 1 Processor -- start 2024-03-25T19:43:07.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.229+0000 7fe88e4e2640 1 -- start start 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.229+0000 7fe88e4e2640 10 monclient: init 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe8880b4f40 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe88813fd90 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe8880b4540 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8880aa0c0 con 0x7fe88813fd90 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8880ab170 con 0x7fe8880b4540 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8880a9ba0 con 0x7fe8880b4f40 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.230 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.230+0000 7fe88e4e2640 10 monclient(hunting): authenticate will time out at 4962.493890s 2024-03-25T19:43:07.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.237+0000 7fe8877fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fe88813fd90 0x7fe8881423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49022/0 (socket says 172.21.15.50:49022) 2024-03-25T19:43:07.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.237+0000 7fe8877fe640 1 -- 172.21.15.50:0/766144503 learned_addr learned my addr 172.21.15.50:0/766144503 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:07.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.251+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731313249 0 0) 0x7fe8880a9ba0 con 0x7fe8880b4f40 2024-03-25T19:43:07.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.251+0000 7fe886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.251+0000 7fe886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.251+0000 7fe886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.251+0000 7fe886ffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:43:07.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.251+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe870003520 con 0x7fe8880b4f40 2024-03-25T19:43:07.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.252+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338032695 0 0) 0x7fe8880ab170 con 0x7fe8880b4540 2024-03-25T19:43:07.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.252+0000 7fe886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.252+0000 7fe886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.252+0000 7fe886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.252+0000 7fe886ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:43:07.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.252+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8880a9ba0 con 0x7fe8880b4540 2024-03-25T19:43:07.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 580820133 0 0) 0x7fe8880a9ba0 con 0x7fe8880b4540 2024-03-25T19:43:07.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 >> v1:172.21.15.50:6790/0 conn(0x7fe8880b4f40 legacy=0x7fe88813f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 >> v1:172.21.15.50:6789/0 conn(0x7fe88813fd90 legacy=0x7fe8881423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 1 monclient: found mon.noname-c 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880a9680 con 0x7fe8880b4540 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe8800019f0 con 0x7fe8880b4540 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe88e4e2640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe88e4e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe886ffd640 10 monclient: dump: 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:07.263 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.263+0000 7fe88e4e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:07.267 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe880001b50 con 0x7fe8880b4540 2024-03-25T19:43:07.267 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe886ffd640 1 -- 172.21.15.50:0/766144503 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe880001f20 con 0x7fe8880b4540 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe886ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe886ffd640 10 monclient: dump: 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:07.268 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:07.269 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.269 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe88e4e2640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:07.269 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe88e4e2640 4 set_mon_vals no callback set 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe88e4e2640 10 monclient: shutdown 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe88e4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe88e4e2640 1 -- 172.21.15.50:0/766144503 >> v1:172.21.15.76:6789/0 conn(0x7fe8880b4540 legacy=0x7fe88814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe88e4e2640 1 -- 172.21.15.50:0/766144503 shutdown_connections 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.268+0000 7fe88e4e2640 1 -- 172.21.15.50:0/766144503 >> 172.21.15.50:0/766144503 conn(0x7fe8880617a0 msgr2=0x7fe888061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.269+0000 7fe88e4e2640 1 -- 172.21.15.50:0/766144503 shutdown_connections 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.269+0000 7fe88e4e2640 1 -- 172.21.15.50:0/766144503 wait complete. 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.269+0000 7fe88e4e2640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.269+0000 7fe88e4e2640 10 monclient: monmap: 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:07.271 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.269+0000 7fe88e4e2640 1 librados: starting msgr at 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.269+0000 7fe88e4e2640 1 librados: starting objecter 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 1 Processor -- start 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 1 -- start start 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 1 librados: setting wanted keys 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 1 librados: calling monclient init 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient: init 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe8880b4540 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe8880b4f40 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe88813fd90 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe888142480 con 0x7fe88813fd90 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88813dd20 con 0x7fe8880b4f40 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe88813e1b0 con 0x7fe8880b4540 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.270+0000 7fe88e4e2640 10 monclient(hunting): authenticate will time out at 4962.534562s 2024-03-25T19:43:07.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.276+0000 7fe8877fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe8880b4f40 0x7fe888138ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:52198/0 (socket says 172.21.15.50:52198) 2024-03-25T19:43:07.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.276+0000 7fe8877fe640 1 -- 172.21.15.50:0/3229035171 learned_addr learned my addr 172.21.15.50:0/3229035171 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:07.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.277+0000 7fe884ff9640 10 client.?.objecter ms_handle_connect 0x7fe8880b4f40 2024-03-25T19:43:07.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.277+0000 7fe884ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.277+0000 7fe884ff9640 10 client.?.objecter ms_handle_connect 0x7fe8880b4540 2024-03-25T19:43:07.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.277+0000 7fe884ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.288 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.289+0000 7fe884ff9640 10 client.?.objecter ms_handle_connect 0x7fe88813fd90 2024-03-25T19:43:07.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.289+0000 7fe884ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.290+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879319661 0 0) 0x7fe88813e1b0 con 0x7fe8880b4540 2024-03-25T19:43:07.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.290+0000 7fe884ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.290+0000 7fe884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.290+0000 7fe884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.290+0000 7fe884ff9640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:43:07.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.290+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe874004eb0 con 0x7fe8880b4540 2024-03-25T19:43:07.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.292+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520626073 0 0) 0x7fe88813dd20 con 0x7fe8880b4f40 2024-03-25T19:43:07.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.292+0000 7fe884ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.292+0000 7fe884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.292+0000 7fe884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.292+0000 7fe884ff9640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:43:07.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.292+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe88813e1b0 con 0x7fe8880b4f40 2024-03-25T19:43:07.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.296+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063515319 0 0) 0x7fe888142480 con 0x7fe88813fd90 2024-03-25T19:43:07.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.296+0000 7fe884ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.296+0000 7fe884ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.296+0000 7fe884ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.296+0000 7fe884ff9640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:43:07.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.296+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe88813dd20 con 0x7fe88813fd90 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 260208096 0 0) 0x7fe88813e1b0 con 0x7fe8880b4f40 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe888142480 con 0x7fe8880b4f40 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe87c001850 con 0x7fe8880b4f40 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.306 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3519415574 0 0) 0x7fe874004eb0 con 0x7fe8880b4540 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe88813e1b0 con 0x7fe8880b4540 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe878001950 con 0x7fe8880b4540 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.306+0000 7fe884ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.307 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.308 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:07.308 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:07.308 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:07.308 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:07.308 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.310+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3045592427 0 0) 0x7fe88813dd20 con 0x7fe88813fd90 2024-03-25T19:43:07.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.310+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe874004eb0 con 0x7fe88813fd90 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.310+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe880001e60 con 0x7fe88813fd90 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.310+0000 7fe884ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.310+0000 7fe884ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.310+0000 7fe884ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:07.310 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1978577313 0 0) 0x7fe888142480 con 0x7fe8880b4f40 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 >> v1:172.21.15.50:6790/0 conn(0x7fe8880b4540 legacy=0x7fe888141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 >> v1:172.21.15.50:6789/0 conn(0x7fe88813fd90 legacy=0x7fe8881d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 1 monclient: found mon.b 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8880b1340 con 0x7fe8880b4f40 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe884ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 5 monclient: authenticate success, global_id 4312 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 10 monclient: _renew_subs 2024-03-25T19:43:07.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:07.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe888139700 con 0x7fe8880b4f40 2024-03-25T19:43:07.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:07.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 10 monclient: _renew_subs 2024-03-25T19:43:07.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:07.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe888140640 con 0x7fe8880b4f40 2024-03-25T19:43:07.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.325+0000 7fe88e4e2640 1 librados: init done 2024-03-25T19:43:07.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.326+0000 7fe88e4e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.326+0000 7fe88e4e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.326+0000 7fe88e4e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:07.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.326+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8880627b0 con 0x7fe8880b4f40 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.632+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe87c001c80 con 0x7fe8880b4f40 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.632+0000 7fe884ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.632+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe87c004980 con 0x7fe8880b4f40 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.632+0000 7fe884ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.632+0000 7fe884ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.632+0000 7fe884ff9640 10 monclient: dump: 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:07.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.632+0000 7fe8867fc640 4 set_mon_vals no callback set 2024-03-25T19:43:07.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.633+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fe87c01aa10 con 0x7fe8880b4f40 2024-03-25T19:43:07.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.633+0000 7fe887fff640 10 monclient: get_auth_request con 0x7fe87406e0f0 auth_method 0 2024-03-25T19:43:07.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.634+0000 7fe884ff9640 10 client.4312.objecter ms_handle_connect 0x7fe87406e0f0 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fe87c054710 con 0x7fe8880b4f40 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 10 client.4312.objecter ms_dispatch 0x7fe888000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe87c050050 con 0x7fe8880b4f40 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.638+0000 7fe884ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:07.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.895+0000 7fe88e4e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:07.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.895+0000 7fe88e4e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:07.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.895+0000 7fe88e4e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:07.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.895+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe888139890 con 0x7fe8880b4f40 2024-03-25T19:43:07.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.922+0000 7fe884ff9640 1 -- 172.21.15.50:0/3229035171 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fe87c052150 con 0x7fe8880b4f40 2024-03-25T19:43:07.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.922+0000 7fe884ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:07.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.922+0000 7fe884ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:07.923 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:07.923 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:07.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 10 librados: watch_flush enter 2024-03-25T19:43:07.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 10 librados: watch_flush exit 2024-03-25T19:43:07.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T19:43:07.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 10 client.4312.objecter successfully canceled tick 2024-03-25T19:43:07.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 >> v1:172.21.15.76:6800/2572556811 conn(0x7fe87406e0f0 legacy=0x7fe8740705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 10 monclient: shutdown 2024-03-25T19:43:07.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:07.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 >> v1:172.21.15.76:6789/0 conn(0x7fe8880b4f40 legacy=0x7fe888138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 shutdown_connections 2024-03-25T19:43:07.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 >> 172.21.15.50:0/3229035171 conn(0x7fe8880617a0 msgr2=0x7fe888061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:07.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 shutdown_connections 2024-03-25T19:43:07.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 1 -- 172.21.15.50:0/3229035171 wait complete. 2024-03-25T19:43:07.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:07.925+0000 7fe88e4e2640 1 librados: shutdown 2024-03-25T19:43:07.937 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:14.296 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:34.802 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.350 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:34.856 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.380 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:34.887 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.406 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:34.877 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.420 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:34.891 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.422 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:34.928 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.425 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:34.931 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.446 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:34.917 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.450 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:34.956 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:14.744 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.859+0000 7eff11f9d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.859+0000 7eff11f9d640 10 monclient: build_initial_monmap 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.859+0000 7eff11f9d640 10 monclient: monmap: 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:43:14.860210+0000 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:43:14.860210+0000 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:43:14.859 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:43:14.860 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.860+0000 7eff11f9d640 1 Processor -- start 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.860+0000 7eff11f9d640 1 -- start start 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.860+0000 7eff11f9d640 10 monclient: init 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 10 monclient(hunting): picked mon.noname-c con 0x7eff0c0a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 10 monclient(hunting): picked mon.noname-a con 0x7eff0c13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 10 monclient(hunting): picked mon.noname-b con 0x7eff0c1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c142520 con 0x7eff0c13f710 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c0b28a0 con 0x7eff0c0a98f0 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c0b2a30 con 0x7eff0c1430c0 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:14.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.861+0000 7eff11f9d640 10 monclient(hunting): authenticate will time out at 4970.124931s 2024-03-25T19:43:14.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.871+0000 7eff0b7fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7eff0c0a98f0 0x7eff0c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:37772/0 (socket says 172.21.15.50:37772) 2024-03-25T19:43:14.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.871+0000 7eff0b7fe640 1 -- 172.21.15.50:0/908537098 learned_addr learned my addr 172.21.15.50:0/908537098 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:14.885 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.885+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810452993 0 0) 0x7eff0c0b28a0 con 0x7eff0c0a98f0 2024-03-25T19:43:14.885 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.885+0000 7eff0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.885 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.885+0000 7eff0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.885 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.885+0000 7eff0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.885 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.885+0000 7eff0a7fc640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:43:14.885 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.885+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef4003520 con 0x7eff0c0a98f0 2024-03-25T19:43:14.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.894+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550057768 0 0) 0x7eff0c142520 con 0x7eff0c13f710 2024-03-25T19:43:14.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.895+0000 7eff0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.895+0000 7eff0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.895+0000 7eff0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.895+0000 7eff0a7fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:43:14.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.895+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff0c0b28a0 con 0x7eff0c13f710 2024-03-25T19:43:14.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210510339 0 0) 0x7efef4003520 con 0x7eff0c0a98f0 2024-03-25T19:43:14.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 >> v1:172.21.15.50:6790/0 conn(0x7eff0c1430c0 legacy=0x7eff0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 >> v1:172.21.15.50:6789/0 conn(0x7eff0c13f710 legacy=0x7eff0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 1 monclient: found mon.noname-c 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c0b44c0 con 0x7eff0c0a98f0 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:14.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efefc001690 con 0x7eff0c0a98f0 2024-03-25T19:43:14.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:14.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff0a7fc640 10 monclient: dump: 2024-03-25T19:43:14.897 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:14.897 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:14.897 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:14.897 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff11f9d640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:43:14.898 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.896+0000 7eff11f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:14.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efefc0017f0 con 0x7eff0c0a98f0 2024-03-25T19:43:14.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:14.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff11f9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:14.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff11f9d640 4 set_mon_vals no callback set 2024-03-25T19:43:14.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff0a7fc640 1 -- 172.21.15.50:0/908537098 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efefc001c00 con 0x7eff0c0a98f0 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff0a7fc640 10 monclient: dump: 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:14.942 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:14.944 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff11f9d640 10 monclient: shutdown 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff11f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.942+0000 7eff11f9d640 1 -- 172.21.15.50:0/908537098 >> v1:172.21.15.76:6789/0 conn(0x7eff0c0a98f0 legacy=0x7eff0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.943+0000 7eff11f9d640 1 -- 172.21.15.50:0/908537098 shutdown_connections 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.943+0000 7eff11f9d640 1 -- 172.21.15.50:0/908537098 >> 172.21.15.50:0/908537098 conn(0x7eff0c0b7ea0 msgr2=0x7eff0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.943+0000 7eff11f9d640 1 -- 172.21.15.50:0/908537098 shutdown_connections 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.943+0000 7eff11f9d640 1 -- 172.21.15.50:0/908537098 wait complete. 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.943+0000 7eff11f9d640 10 monclient: build_initial_monmap 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.943+0000 7eff11f9d640 10 monclient: monmap: 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:43:14.945 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.944+0000 7eff11f9d640 1 librados: starting msgr at 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.944+0000 7eff11f9d640 1 librados: starting objecter 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.944+0000 7eff11f9d640 1 Processor -- start 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.944+0000 7eff11f9d640 1 -- start start 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.944+0000 7eff11f9d640 1 librados: setting wanted keys 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.944+0000 7eff11f9d640 1 librados: calling monclient init 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.944+0000 7eff11f9d640 10 monclient: init 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 10 monclient(hunting): picked mon.noname-c con 0x7eff0c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 10 monclient(hunting): picked mon.noname-a con 0x7eff0c13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:14.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 10 monclient(hunting): picked mon.noname-b con 0x7eff0c1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:14.947 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c1d7380 con 0x7eff0c13f710 2024-03-25T19:43:14.947 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c139840 con 0x7eff0c1430c0 2024-03-25T19:43:14.947 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c1d8200 con 0x7eff0c0a98f0 2024-03-25T19:43:14.947 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:14.947 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.945+0000 7eff11f9d640 10 monclient(hunting): authenticate will time out at 4970.208988s 2024-03-25T19:43:14.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.956+0000 7eff0bfff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7eff0c1430c0 0x7eff0c1daf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:37818/0 (socket says 172.21.15.50:37818) 2024-03-25T19:43:14.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.956+0000 7eff0bfff640 1 -- 172.21.15.50:0/1943097233 learned_addr learned my addr 172.21.15.50:0/1943097233 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:14.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.957+0000 7efef3fff640 10 client.?.objecter ms_handle_connect 0x7eff0c1430c0 2024-03-25T19:43:14.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.957+0000 7efef3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:14.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.959+0000 7efef3fff640 10 client.?.objecter ms_handle_connect 0x7eff0c0a98f0 2024-03-25T19:43:14.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.959+0000 7efef3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:14.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.959+0000 7efef3fff640 10 client.?.objecter ms_handle_connect 0x7eff0c13f710 2024-03-25T19:43:14.959 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.959+0000 7efef3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:14.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.970+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688490209 0 0) 0x7eff0c1d8200 con 0x7eff0c0a98f0 2024-03-25T19:43:14.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.970+0000 7efef3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.970+0000 7efef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.970+0000 7efef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.970+0000 7efef3fff640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:43:14.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.970+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef8004eb0 con 0x7eff0c0a98f0 2024-03-25T19:43:14.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931463256 0 0) 0x7eff0c139840 con 0x7eff0c1430c0 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff0c1d8200 con 0x7eff0c1430c0 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675925388 0 0) 0x7eff0c1d7380 con 0x7eff0c13f710 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:43:14.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.973+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff0c139840 con 0x7eff0c13f710 2024-03-25T19:43:14.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.984+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2430275565 0 0) 0x7eff0c139840 con 0x7eff0c13f710 2024-03-25T19:43:14.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.984+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff0c1d7380 con 0x7eff0c13f710 2024-03-25T19:43:14.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.984+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7eff00001930 con 0x7eff0c13f710 2024-03-25T19:43:14.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.984+0000 7efef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.984+0000 7efef3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.984+0000 7efef3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:14.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.984+0000 7efef3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:14.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.988+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1439173020 0 0) 0x7eff0c1d8200 con 0x7eff0c1430c0 2024-03-25T19:43:14.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.988+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff0c139840 con 0x7eff0c1430c0 2024-03-25T19:43:14.999 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.988+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7eff04001c10 con 0x7eff0c1430c0 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.988+0000 7efef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.988+0000 7efef3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.988+0000 7efef3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.989+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3685337015 0 0) 0x7efef8004eb0 con 0x7eff0c0a98f0 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.989+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff0c1d8200 con 0x7eff0c0a98f0 2024-03-25T19:43:15.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.989+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efefc001b00 con 0x7eff0c0a98f0 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.989+0000 7efef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.989+0000 7efef3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:14.989+0000 7efef3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:15.001 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:15.009 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.009+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604523603 0 0) 0x7eff0c1d8200 con 0x7eff0c0a98f0 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 >> v1:172.21.15.76:6789/0 conn(0x7eff0c1430c0 legacy=0x7eff0c1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 >> v1:172.21.15.50:6789/0 conn(0x7eff0c13f710 legacy=0x7eff0c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 1 monclient: found mon.c 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c1d75d0 con 0x7eff0c0a98f0 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7efef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 5 monclient: authenticate success, global_id 4319 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 10 monclient: _renew_subs 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:15.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c1db6a0 con 0x7eff0c0a98f0 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 10 monclient: _renew_subs 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c137910 con 0x7eff0c0a98f0 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.010+0000 7eff11f9d640 1 librados: init done 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.011+0000 7eff11f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.011+0000 7eff11f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.011+0000 7eff11f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:15.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.011+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c0627b0 con 0x7eff0c0a98f0 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.330+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efefc001220 con 0x7eff0c0a98f0 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.330+0000 7efef3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.330+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efefc004c30 con 0x7eff0c0a98f0 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.330+0000 7efef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.330+0000 7efef3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.330+0000 7efef3fff640 10 monclient: dump: 2024-03-25T19:43:15.330 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:15.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.330+0000 7eff09ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.331+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7efefc01ac00 con 0x7eff0c0a98f0 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.332+0000 7eff0affd640 10 monclient: get_auth_request con 0x7efef806e160 auth_method 0 2024-03-25T19:43:15.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.332+0000 7efef3fff640 10 client.4319.objecter ms_handle_connect 0x7efef806e160 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7efefc054840 con 0x7eff0c0a98f0 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 10 client.4319.objecter ms_dispatch 0x7eff0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efefc050180 con 0x7eff0c0a98f0 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.338+0000 7efef3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:15.575 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.576+0000 7eff11f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.575 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.576+0000 7eff11f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.575 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.576+0000 7eff11f9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:15.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.576+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff0c1d7810 con 0x7eff0c0a98f0 2024-03-25T19:43:15.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.622+0000 7efef3fff640 1 -- 172.21.15.50:0/1943097233 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7efefc052280 con 0x7eff0c0a98f0 2024-03-25T19:43:15.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.622+0000 7efef3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.622+0000 7efef3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:15.621 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:15.621 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:15.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 10 librados: watch_flush enter 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 10 librados: watch_flush exit 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 10 client.4319.objecter successfully canceled tick 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 >> v1:172.21.15.76:6800/2572556811 conn(0x7efef806e160 legacy=0x7efef8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 10 monclient: shutdown 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 >> v1:172.21.15.50:6790/0 conn(0x7eff0c0a98f0 legacy=0x7eff0c1d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 shutdown_connections 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.624+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 >> 172.21.15.50:0/1943097233 conn(0x7eff0c0b7ea0 msgr2=0x7eff0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.625+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 shutdown_connections 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.625+0000 7eff11f9d640 1 -- 172.21.15.50:0/1943097233 wait complete. 2024-03-25T19:43:15.625 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:15.625+0000 7eff11f9d640 1 librados: shutdown 2024-03-25T19:43:15.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:22.442 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.556+0000 7f4e20e45640 10 monclient: get_monmap_and_config 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.556+0000 7f4e20e45640 10 monclient: build_initial_monmap 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.556+0000 7f4e20e45640 10 monclient: monmap: 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:43:22.557672+0000 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:43:22.557672+0000 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:22.556 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:22.557 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:22.557 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:43:22.557 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:43:22.557 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:22.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.557+0000 7f4e20e45640 1 Processor -- start 2024-03-25T19:43:22.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.557+0000 7f4e20e45640 1 -- start start 2024-03-25T19:43:22.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.557+0000 7f4e20e45640 10 monclient: init 2024-03-25T19:43:22.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:22.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:22.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 10 monclient(hunting): picked mon.noname-a con 0x7f4e1c0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:22.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 10 monclient(hunting): picked mon.noname-b con 0x7f4e1c13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:22.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 10 monclient(hunting): picked mon.noname-c con 0x7f4e1c1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:22.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c142520 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c0b28a0 con 0x7f4e1c1430c0 2024-03-25T19:43:22.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c0b2a30 con 0x7f4e1c13f710 2024-03-25T19:43:22.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:22.559 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.558+0000 7f4e20e45640 10 monclient(hunting): authenticate will time out at 4977.822115s 2024-03-25T19:43:22.561 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.562+0000 7f4e13fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f4e1c13f710 0x7f4e1c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53078/0 (socket says 172.21.15.50:53078) 2024-03-25T19:43:22.561 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.562+0000 7f4e13fff640 1 -- 172.21.15.50:0/323919964 learned_addr learned my addr 172.21.15.50:0/323919964 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:22.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.578+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306917010 0 0) 0x7f4e1c142520 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.578+0000 7f4e19d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.578+0000 7f4e19d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.578+0000 7f4e19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.578+0000 7f4e19d9b640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:43:22.578 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.578+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e08003520 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.587+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304840286 0 0) 0x7f4e1c0b28a0 con 0x7f4e1c1430c0 2024-03-25T19:43:22.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.587+0000 7f4e19d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.587+0000 7f4e19d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.587+0000 7f4e19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.587+0000 7f4e19d9b640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:43:22.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.587+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c142520 con 0x7f4e1c1430c0 2024-03-25T19:43:22.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.588+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776078325 0 0) 0x7f4e1c0b2a30 con 0x7f4e1c13f710 2024-03-25T19:43:22.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.588+0000 7f4e19d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.588+0000 7f4e19d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.588+0000 7f4e19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.588+0000 7f4e19d9b640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:43:22.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.588+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c0b28a0 con 0x7f4e1c13f710 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 427283139 0 0) 0x7f4e1c0b28a0 con 0x7f4e1c13f710 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 >> v1:172.21.15.76:6789/0 conn(0x7f4e1c1430c0 legacy=0x7f4e1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 >> v1:172.21.15.50:6789/0 conn(0x7f4e1c0a98f0 legacy=0x7f4e1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 1 monclient: found mon.noname-b 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:43:22.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c0b44c0 con 0x7f4e1c13f710 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e040015b0 con 0x7f4e1c13f710 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e19d9b640 10 monclient: dump: 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:22.603 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:22.604 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:22.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e20e45640 5 monclient: authenticate success, global_id 4322 2024-03-25T19:43:22.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.602+0000 7f4e20e45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e04001710 con 0x7f4e1c13f710 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e19d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e19d9b640 1 -- 172.21.15.50:0/323919964 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e04001b60 con 0x7f4e1c13f710 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e19d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e19d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e19d9b640 10 monclient: dump: 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:22.612 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:22.613 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:22.613 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:22.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e20e45640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:22.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.612+0000 7f4e20e45640 4 set_mon_vals no callback set 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.613+0000 7f4e20e45640 10 monclient: shutdown 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.613+0000 7f4e20e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.613+0000 7f4e20e45640 1 -- 172.21.15.50:0/323919964 >> v1:172.21.15.50:6790/0 conn(0x7f4e1c13f710 legacy=0x7f4e1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.613+0000 7f4e20e45640 1 -- 172.21.15.50:0/323919964 shutdown_connections 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.613+0000 7f4e20e45640 1 -- 172.21.15.50:0/323919964 >> 172.21.15.50:0/323919964 conn(0x7f4e1c0b7ea0 msgr2=0x7f4e1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.613+0000 7f4e20e45640 1 -- 172.21.15.50:0/323919964 shutdown_connections 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.613+0000 7f4e20e45640 1 -- 172.21.15.50:0/323919964 wait complete. 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.614+0000 7f4e20e45640 10 monclient: build_initial_monmap 2024-03-25T19:43:22.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.614+0000 7f4e20e45640 10 monclient: monmap: 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 librados: starting msgr at 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 librados: starting objecter 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 Processor -- start 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 -- start start 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 librados: setting wanted keys 2024-03-25T19:43:22.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 librados: calling monclient init 2024-03-25T19:43:22.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient: init 2024-03-25T19:43:22.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient(hunting): picked mon.noname-c con 0x7f4e1c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient(hunting): picked mon.noname-b con 0x7f4e1c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient(hunting): picked mon.noname-a con 0x7f4e1c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c0abd30 con 0x7f4e1c1430c0 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c1d84b0 con 0x7f4e1c13f710 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1c1d7880 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:22.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.615+0000 7f4e20e45640 10 monclient(hunting): authenticate will time out at 4977.879565s 2024-03-25T19:43:22.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.631+0000 7f4e1a59c640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f4e1c0a98f0 0x7f4e1c1d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53128/0 (socket says 172.21.15.50:53128) 2024-03-25T19:43:22.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.631+0000 7f4e1a59c640 1 -- 172.21.15.50:0/2068403998 learned_addr learned my addr 172.21.15.50:0/2068403998 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:22.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.634+0000 7f4e12ffd640 10 client.?.objecter ms_handle_connect 0x7f4e1c0a98f0 2024-03-25T19:43:22.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.634+0000 7f4e12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:22.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.635+0000 7f4e12ffd640 10 client.?.objecter ms_handle_connect 0x7f4e1c13f710 2024-03-25T19:43:22.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.635+0000 7f4e12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:22.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.639+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010670048 0 0) 0x7f4e1c1d7880 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.639+0000 7f4e12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.639+0000 7f4e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.639+0000 7f4e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.639+0000 7f4e12ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:43:22.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.639+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dfc004eb0 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.640+0000 7f4e12ffd640 10 client.?.objecter ms_handle_connect 0x7f4e1c1430c0 2024-03-25T19:43:22.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.640+0000 7f4e12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:22.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.641+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585270051 0 0) 0x7f4e1c1d84b0 con 0x7f4e1c13f710 2024-03-25T19:43:22.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.641+0000 7f4e12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.641+0000 7f4e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.641+0000 7f4e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.641+0000 7f4e12ffd640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:43:22.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.641+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c1d7880 con 0x7f4e1c13f710 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.647+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 744292120 0 0) 0x7f4dfc004eb0 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.647+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e1c1d84b0 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.647+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e0c0019b0 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.647+0000 7f4e12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.647+0000 7f4e12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.647+0000 7f4e12ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:22.647 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:22.648 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:22.648 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:22.648 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:22.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.647+0000 7f4e12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.649+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1927485672 0 0) 0x7f4e1c1d7880 con 0x7f4e1c13f710 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.649+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dfc004eb0 con 0x7f4e1c13f710 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.649+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e04001a20 con 0x7f4e1c13f710 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.649+0000 7f4e12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.649+0000 7f4e12ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.649+0000 7f4e12ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:22.649 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:22.650 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:22.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.652+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314771165 0 0) 0x7f4e1c0abd30 con 0x7f4e1c1430c0 2024-03-25T19:43:22.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.652+0000 7f4e12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.652+0000 7f4e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.652+0000 7f4e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.652+0000 7f4e12ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:43:22.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.652+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e1c1d7880 con 0x7f4e1c1430c0 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1366976472 0 0) 0x7f4e1c1d84b0 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 >> v1:172.21.15.76:6789/0 conn(0x7f4e1c13f710 legacy=0x7f4e1c138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 >> v1:172.21.15.50:6789/0 conn(0x7f4e1c1430c0 legacy=0x7f4e1c1d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 1 monclient: found mon.c 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c138e60 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:22.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 5 monclient: authenticate success, global_id 4325 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 10 monclient: _renew_subs 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c1d50a0 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 10 monclient: _renew_subs 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c138170 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.660+0000 7f4e20e45640 1 librados: init done 2024-03-25T19:43:22.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.661+0000 7f4e20e45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:22.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.661+0000 7f4e20e45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:22.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.661+0000 7f4e20e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:22.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.661+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1c1d7a10 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.995+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e0c003610 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.995+0000 7f4e12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.995+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e0c0039e0 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.995+0000 7f4e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.995+0000 7f4e12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.995+0000 7f4e12ffd640 10 monclient: dump: 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:22.995 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:22.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.995+0000 7f4e1959a640 4 set_mon_vals no callback set 2024-03-25T19:43:22.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.996+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f4e0c003c00 con 0x7f4e1c0a98f0 2024-03-25T19:43:22.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.997+0000 7f4e13fff640 10 monclient: get_auth_request con 0x7f4dfc06df50 auth_method 0 2024-03-25T19:43:22.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:22.997+0000 7f4e12ffd640 10 client.4325.objecter ms_handle_connect 0x7f4dfc06df50 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f4e0c0547f0 con 0x7f4e1c0a98f0 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 10 client.4325.objecter ms_dispatch 0x7f4e1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e0c050300 con 0x7f4e1c0a98f0 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.003+0000 7f4e12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:23.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.272+0000 7f4e20e45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.272+0000 7f4e20e45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.272+0000 7f4e20e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:23.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.272+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e1c074cc0 con 0x7f4e1c0a98f0 2024-03-25T19:43:23.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.308+0000 7f4e12ffd640 1 -- 172.21.15.50:0/2068403998 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f4e0c052230 con 0x7f4e1c0a98f0 2024-03-25T19:43:23.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.308+0000 7f4e12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.308 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.308+0000 7f4e12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:23.308 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:23.308 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:23.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 10 librados: watch_flush enter 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 10 librados: watch_flush exit 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 10 client.4325.objecter successfully canceled tick 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 >> v1:172.21.15.76:6800/2572556811 conn(0x7f4dfc06df50 legacy=0x7f4dfc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 10 monclient: shutdown 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 >> v1:172.21.15.50:6790/0 conn(0x7f4e1c0a98f0 legacy=0x7f4e1c1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 shutdown_connections 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.310+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 >> 172.21.15.50:0/2068403998 conn(0x7f4e1c0b7ea0 msgr2=0x7f4e1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.311+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 shutdown_connections 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.311+0000 7f4e20e45640 1 -- 172.21.15.50:0/2068403998 wait complete. 2024-03-25T19:43:23.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:23.311+0000 7f4e20e45640 1 librados: shutdown 2024-03-25T19:43:23.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:30.128 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.241+0000 7f38b699d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.241+0000 7f38b699d640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.241+0000 7f38b699d640 10 monclient: monmap: 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:43:30.242263+0000 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:43:30.242263+0000 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:43:30.241 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.242+0000 7f38b699d640 1 Processor -- start 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.242+0000 7f38b699d640 1 -- start start 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.242+0000 7f38b699d640 10 monclient: init 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.242+0000 7f38b699d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.242+0000 7f38b699d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.242+0000 7f38b699d640 10 monclient(hunting): picked mon.noname-a con 0x7f38b00a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.242+0000 7f38b699d640 10 monclient(hunting): picked mon.noname-c con 0x7f38b013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.243+0000 7f38b699d640 10 monclient(hunting): picked mon.noname-b con 0x7f38b01430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.243+0000 7f38b699d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b0142520 con 0x7f38b00a98f0 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.243+0000 7f38b699d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b00b28a0 con 0x7f38b013f710 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.243+0000 7f38b699d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b00b2a30 con 0x7f38b01430c0 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.243+0000 7f38b699d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.243 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.243+0000 7f38b699d640 10 monclient(hunting): authenticate will time out at 4985.506749s 2024-03-25T19:43:30.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.252+0000 7f38affff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f38b00a98f0 0x7f38b00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53478/0 (socket says 172.21.15.50:53478) 2024-03-25T19:43:30.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.252+0000 7f38affff640 1 -- 172.21.15.50:0/3924638997 learned_addr learned my addr 172.21.15.50:0/3924638997 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:30.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.260+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935084259 0 0) 0x7f38b0142520 con 0x7f38b00a98f0 2024-03-25T19:43:30.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.260+0000 7f38aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.260+0000 7f38aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.260+0000 7f38aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.260+0000 7f38aeffd640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:43:30.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.260+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3898003520 con 0x7f38b00a98f0 2024-03-25T19:43:30.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.278+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318494044 0 0) 0x7f38b00b28a0 con 0x7f38b013f710 2024-03-25T19:43:30.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.278+0000 7f38aeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.278+0000 7f38aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.278+0000 7f38aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.278+0000 7f38aeffd640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:43:30.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.278+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b0142520 con 0x7f38b013f710 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703900581 0 0) 0x7f3898003520 con 0x7f38b00a98f0 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 >> v1:172.21.15.50:6790/0 conn(0x7f38b01430c0 legacy=0x7f38b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 >> v1:172.21.15.76:6789/0 conn(0x7f38b013f710 legacy=0x7f38b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 1 monclient: found mon.noname-a 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b00b44c0 con 0x7f38b00a98f0 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f38a0001690 con 0x7f38b00a98f0 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38aeffd640 10 monclient: dump: 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38b699d640 5 monclient: authenticate success, global_id 4344 2024-03-25T19:43:30.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.284+0000 7f38b699d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38a00017f0 con 0x7f38b00a98f0 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38aeffd640 1 -- 172.21.15.50:0/3924638997 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f38a0001c00 con 0x7f38b00a98f0 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38b699d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38b699d640 4 set_mon_vals no callback set 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.294+0000 7f38aeffd640 10 monclient: dump: 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.295 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.295+0000 7f38b699d640 10 monclient: shutdown 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.295+0000 7f38b699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.295+0000 7f38b699d640 1 -- 172.21.15.50:0/3924638997 >> v1:172.21.15.50:6789/0 conn(0x7f38b00a98f0 legacy=0x7f38b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.295+0000 7f38b699d640 1 -- 172.21.15.50:0/3924638997 shutdown_connections 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.295+0000 7f38b699d640 1 -- 172.21.15.50:0/3924638997 >> 172.21.15.50:0/3924638997 conn(0x7f38b00b7ea0 msgr2=0x7f38b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.295+0000 7f38b699d640 1 -- 172.21.15.50:0/3924638997 shutdown_connections 2024-03-25T19:43:30.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.295+0000 7f38b699d640 1 -- 172.21.15.50:0/3924638997 wait complete. 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.296+0000 7f38b699d640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.296+0000 7f38b699d640 10 monclient: monmap: 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:43:30.298 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.297+0000 7f38b699d640 1 librados: starting msgr at 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.297+0000 7f38b699d640 1 librados: starting objecter 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.297+0000 7f38b699d640 1 Processor -- start 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.297+0000 7f38b699d640 1 -- start start 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.297+0000 7f38b699d640 1 librados: setting wanted keys 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.297+0000 7f38b699d640 1 librados: calling monclient init 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.297+0000 7f38b699d640 10 monclient: init 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 10 monclient(hunting): picked mon.noname-c con 0x7f38b00a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 10 monclient(hunting): picked mon.noname-a con 0x7f38b013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 10 monclient(hunting): picked mon.noname-b con 0x7f38b01430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:30.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b00b7ce0 con 0x7f38b013f710 2024-03-25T19:43:30.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b01319b0 con 0x7f38b01430c0 2024-03-25T19:43:30.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b00b7800 con 0x7f38b00a98f0 2024-03-25T19:43:30.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.298+0000 7f38b699d640 10 monclient(hunting): authenticate will time out at 4985.562172s 2024-03-25T19:43:30.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.304+0000 7f38affff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f38b00a98f0 0x7f38b01dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:55418/0 (socket says 172.21.15.50:55418) 2024-03-25T19:43:30.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.304+0000 7f38affff640 1 -- 172.21.15.50:0/4034155062 learned_addr learned my addr 172.21.15.50:0/4034155062 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:30.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.308+0000 7f38acff9640 10 client.?.objecter ms_handle_connect 0x7f38b00a98f0 2024-03-25T19:43:30.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.308+0000 7f38acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.309+0000 7f38acff9640 10 client.?.objecter ms_handle_connect 0x7f38b013f710 2024-03-25T19:43:30.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.309+0000 7f38acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.316+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618175216 0 0) 0x7f38b00b7ce0 con 0x7f38b013f710 2024-03-25T19:43:30.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.316+0000 7f38acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.316+0000 7f38acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.316+0000 7f38acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.316+0000 7f38acff9640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:43:30.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.316+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c004eb0 con 0x7f38b013f710 2024-03-25T19:43:30.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.318+0000 7f38acff9640 10 client.?.objecter ms_handle_connect 0x7f38b01430c0 2024-03-25T19:43:30.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.318+0000 7f38acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.326+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289969055 0 0) 0x7f38b00b7800 con 0x7f38b00a98f0 2024-03-25T19:43:30.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.326+0000 7f38acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.326+0000 7f38acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.326+0000 7f38acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.326+0000 7f38acff9640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:43:30.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.326+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b00b7ce0 con 0x7f38b00a98f0 2024-03-25T19:43:30.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.337+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121007473 0 0) 0x7f38b01319b0 con 0x7f38b01430c0 2024-03-25T19:43:30.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.337+0000 7f38acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.337+0000 7f38acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.337+0000 7f38acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.337+0000 7f38acff9640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:43:30.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.337+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b00b7800 con 0x7f38b01430c0 2024-03-25T19:43:30.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.339+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1827674330 0 0) 0x7f389c004eb0 con 0x7f38b013f710 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.339+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38b01319b0 con 0x7f38b013f710 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.340+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f38a4001930 con 0x7f38b013f710 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.340+0000 7f38acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.340+0000 7f38acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.340+0000 7f38acff9640 10 monclient(hunting): dump: 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:30.340 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.340+0000 7f38acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:30.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.355+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4084053304 0 0) 0x7f38b00b7800 con 0x7f38b01430c0 2024-03-25T19:43:30.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.355+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f389c004eb0 con 0x7f38b01430c0 2024-03-25T19:43:30.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.355+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f38a8001a90 con 0x7f38b01430c0 2024-03-25T19:43:30.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.355+0000 7f38acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.355+0000 7f38acff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.355+0000 7f38acff9640 10 monclient(hunting): dump: 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:30.355 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.359+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165729532 0 0) 0x7f38b00b7ce0 con 0x7f38b00a98f0 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.359+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38b00b7800 con 0x7f38b00a98f0 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.359+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f38a0001b40 con 0x7f38b00a98f0 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.359+0000 7f38acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.359+0000 7f38acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.359+0000 7f38acff9640 10 monclient(hunting): dump: 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:30.359 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:30.360 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 282068340 0 0) 0x7f38b01319b0 con 0x7f38b013f710 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 >> v1:172.21.15.50:6790/0 conn(0x7f38b00a98f0 legacy=0x7f38b01dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 >> v1:172.21.15.76:6789/0 conn(0x7f38b01430c0 legacy=0x7f38b01312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 1 monclient: found mon.a 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b0136f50 con 0x7f38b013f710 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 10 monclient: _renew_subs 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b01350f0 con 0x7f38b013f710 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 10 monclient: _renew_subs 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b01353a0 con 0x7f38b013f710 2024-03-25T19:43:30.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.370+0000 7f38b699d640 1 librados: init done 2024-03-25T19:43:30.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.371+0000 7f38b699d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.371+0000 7f38b699d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.371+0000 7f38b699d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:30.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.371+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b00627b0 con 0x7f38b013f710 2024-03-25T19:43:30.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.692+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38a4003610 con 0x7f38b013f710 2024-03-25T19:43:30.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.692+0000 7f38acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.692+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f38a4003920 con 0x7f38b013f710 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.692+0000 7f38acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.692+0000 7f38acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.692+0000 7f38acff9640 10 monclient: dump: 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:30.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.692+0000 7f38ae7fc640 4 set_mon_vals no callback set 2024-03-25T19:43:30.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.693+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f38a40198f0 con 0x7f38b013f710 2024-03-25T19:43:30.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.693+0000 7f38affff640 10 monclient: get_auth_request con 0x7f389c06e0f0 auth_method 0 2024-03-25T19:43:30.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.693+0000 7f38acff9640 10 client.4347.objecter ms_handle_connect 0x7f389c06e0f0 2024-03-25T19:43:30.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f38a40535b0 con 0x7f38b013f710 2024-03-25T19:43:30.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 10 client.4347.objecter ms_dispatch 0x7f38b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:30.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:30.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:30.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T19:43:30.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38a404eef0 con 0x7f38b013f710 2024-03-25T19:43:30.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.703+0000 7f38acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:30.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.945+0000 7f38b699d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:30.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.945+0000 7f38b699d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:30.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.945+0000 7f38b699d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:30.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.945+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38b013e270 con 0x7f38b013f710 2024-03-25T19:43:30.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.971+0000 7f38acff9640 1 -- 172.21.15.50:0/4034155062 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f38a4050ff0 con 0x7f38b013f710 2024-03-25T19:43:30.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.971+0000 7f38acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:30.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.971+0000 7f38acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:30.971 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:30.971 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:30.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 10 librados: watch_flush enter 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 10 librados: watch_flush exit 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 10 client.4347.objecter successfully canceled tick 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 >> v1:172.21.15.76:6800/2572556811 conn(0x7f389c06e0f0 legacy=0x7f389c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 10 monclient: shutdown 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.973+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 >> v1:172.21.15.50:6789/0 conn(0x7f38b013f710 legacy=0x7f38b0130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.974+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 shutdown_connections 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.974+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 >> 172.21.15.50:0/4034155062 conn(0x7f38b00b7ea0 msgr2=0x7f38b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.974+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 shutdown_connections 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.974+0000 7f38b699d640 1 -- 172.21.15.50:0/4034155062 wait complete. 2024-03-25T19:43:30.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:30.974+0000 7f38b699d640 1 librados: shutdown 2024-03-25T19:43:30.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:34.375 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:54.881 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.424 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:54.930 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.459 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:02:54.965 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.488 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:54.959 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.490 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:54.996 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.495 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:55.000 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.497 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:54.968 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.531 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:02:55.001 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:34.539 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:02:55.045 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.790 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:37.901 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.901+0000 7f4e5dd1c640 10 monclient: get_monmap_and_config 2024-03-25T19:43:37.901 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.901+0000 7f4e5dd1c640 10 monclient: build_initial_monmap 2024-03-25T19:43:37.901 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.901+0000 7f4e5dd1c640 10 monclient: monmap: 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:43:37.903124+0000 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:43:37.903124+0000 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:43:37.902 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:37.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 1 Processor -- start 2024-03-25T19:43:37.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 1 -- start start 2024-03-25T19:43:37.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 10 monclient: init 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 10 monclient(hunting): picked mon.noname-b con 0x7f4e580b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 10 monclient(hunting): picked mon.noname-c con 0x7f4e580b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e580af940 con 0x7f4e5814be90 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e580b0a70 con 0x7f4e580b1730 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e580b0d50 con 0x7f4e580b4d60 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.902+0000 7f4e5dd1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.903+0000 7f4e5dd1c640 10 monclient(hunting): authenticate will time out at 4993.167704s 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.903+0000 7f4e577fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f4e580b4d60 0x7f4e5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:34492/0 (socket says 172.21.15.50:34492) 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.903+0000 7f4e577fe640 1 -- 172.21.15.50:0/3746024735 learned_addr learned my addr 172.21.15.50:0/3746024735 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:37.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.903+0000 7f4e56ffd640 1 --1- 172.21.15.50:0/3746024735 >> v1:172.21.15.76:6789/0 conn(0x7f4e580b1730 0x7f4e580b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:48398/0 (socket says 172.21.15.50:48398) 2024-03-25T19:43:37.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.987+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898623419 0 0) 0x7f4e580b0a70 con 0x7f4e580b1730 2024-03-25T19:43:37.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.987+0000 7f4e567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:37.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.987+0000 7f4e567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:37.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.987+0000 7f4e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:37.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.987+0000 7f4e567fc640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:43:37.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.987+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e40003520 con 0x7f4e580b1730 2024-03-25T19:43:37.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.996+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175276271 0 0) 0x7f4e580af940 con 0x7f4e5814be90 2024-03-25T19:43:37.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.996+0000 7f4e567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:37.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.996+0000 7f4e567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:37.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.996+0000 7f4e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:37.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.996+0000 7f4e567fc640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:43:37.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:37.996+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e580b0a70 con 0x7f4e5814be90 2024-03-25T19:43:38.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067332166 0 0) 0x7f4e40003520 con 0x7f4e580b1730 2024-03-25T19:43:38.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 >> v1:172.21.15.50:6790/0 conn(0x7f4e580b4d60 legacy=0x7f4e5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 >> v1:172.21.15.50:6789/0 conn(0x7f4e5814be90 legacy=0x7f4e5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 1 monclient: found mon.noname-c 2024-03-25T19:43:38.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e580b1030 con 0x7f4e580b1730 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e4c001690 con 0x7f4e580b1730 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e567fc640 10 monclient: dump: 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.004 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e5dd1c640 5 monclient: authenticate success, global_id 4333 2024-03-25T19:43:38.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.002+0000 7f4e5dd1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:38.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e4c0017f0 con 0x7f4e580b1730 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e567fc640 1 -- 172.21.15.50:0/3746024735 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e4c001c00 con 0x7f4e580b1730 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e567fc640 10 monclient: dump: 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:38.023 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:38.024 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:38.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e5dd1c640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:38.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.022+0000 7f4e5dd1c640 4 set_mon_vals no callback set 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 10 monclient: shutdown 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3746024735 >> v1:172.21.15.76:6789/0 conn(0x7f4e580b1730 legacy=0x7f4e580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3746024735 shutdown_connections 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3746024735 >> 172.21.15.50:0/3746024735 conn(0x7f4e580617a0 msgr2=0x7f4e58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3746024735 shutdown_connections 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3746024735 wait complete. 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 10 monclient: build_initial_monmap 2024-03-25T19:43:38.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.023+0000 7f4e5dd1c640 10 monclient: monmap: 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.024+0000 7f4e5dd1c640 1 librados: starting msgr at 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.024+0000 7f4e5dd1c640 1 librados: starting objecter 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.024+0000 7f4e5dd1c640 1 Processor -- start 2024-03-25T19:43:38.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.024+0000 7f4e5dd1c640 1 -- start start 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.024+0000 7f4e5dd1c640 1 librados: setting wanted keys 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.024+0000 7f4e5dd1c640 1 librados: calling monclient init 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.024+0000 7f4e5dd1c640 10 monclient: init 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 10 monclient(hunting): picked mon.noname-c con 0x7f4e580b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 10 monclient(hunting): picked mon.noname-b con 0x7f4e580b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5813bfb0 con 0x7f4e5814be90 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e581458e0 con 0x7f4e580b4d60 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5812f7b0 con 0x7f4e580b1730 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:38.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.025+0000 7f4e5dd1c640 10 monclient(hunting): authenticate will time out at 4993.289835s 2024-03-25T19:43:38.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.036+0000 7f4e577fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f4e580b1730 0x7f4e5813b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:34536/0 (socket says 172.21.15.50:34536) 2024-03-25T19:43:38.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.036+0000 7f4e577fe640 1 -- 172.21.15.50:0/3903733737 learned_addr learned my addr 172.21.15.50:0/3903733737 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:38.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.038+0000 7f4e3ffff640 10 client.?.objecter ms_handle_connect 0x7f4e580b4d60 2024-03-25T19:43:38.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.038+0000 7f4e3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.039+0000 7f4e3ffff640 10 client.?.objecter ms_handle_connect 0x7f4e580b1730 2024-03-25T19:43:38.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.039+0000 7f4e3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.043+0000 7f4e3ffff640 10 client.?.objecter ms_handle_connect 0x7f4e5814be90 2024-03-25T19:43:38.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.043+0000 7f4e3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.047+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423221534 0 0) 0x7f4e581458e0 con 0x7f4e580b4d60 2024-03-25T19:43:38.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.047+0000 7f4e3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.047+0000 7f4e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.047+0000 7f4e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.047+0000 7f4e3ffff640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:43:38.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.047+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e44004eb0 con 0x7f4e580b4d60 2024-03-25T19:43:38.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.050+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170515081 0 0) 0x7f4e5812f7b0 con 0x7f4e580b1730 2024-03-25T19:43:38.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.050+0000 7f4e3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.050+0000 7f4e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.050+0000 7f4e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.050+0000 7f4e3ffff640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:43:38.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.050+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e581458e0 con 0x7f4e580b1730 2024-03-25T19:43:38.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.058+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177658565 0 0) 0x7f4e5813bfb0 con 0x7f4e5814be90 2024-03-25T19:43:38.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.058+0000 7f4e3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.058+0000 7f4e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.058+0000 7f4e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.058+0000 7f4e3ffff640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:43:38.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.058+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5812f7b0 con 0x7f4e5814be90 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261226267 0 0) 0x7f4e5812f7b0 con 0x7f4e5814be90 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e5813bfb0 con 0x7f4e5814be90 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e50001bd0 con 0x7f4e5814be90 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 10 monclient(hunting): dump: 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:38.071 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200009900 0 0) 0x7f4e44004eb0 con 0x7f4e580b4d60 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e5812f7b0 con 0x7f4e580b4d60 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e4c001bc0 con 0x7f4e580b4d60 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.070+0000 7f4e3ffff640 10 monclient(hunting): dump: 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:38.072 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:38.073 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:38.073 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:38.073 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.087+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1196666716 0 0) 0x7f4e581458e0 con 0x7f4e580b1730 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.087+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e44004eb0 con 0x7f4e580b1730 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.087+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e480017e0 con 0x7f4e580b1730 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.087+0000 7f4e3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.087+0000 7f4e3ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.087+0000 7f4e3ffff640 10 monclient(hunting): dump: 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:38.088 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.089 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.089 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.089 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:38.089 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:38.089 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:38.089 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:38.089 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 387018589 0 0) 0x7f4e5812f7b0 con 0x7f4e580b4d60 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 >> v1:172.21.15.50:6790/0 conn(0x7f4e580b1730 legacy=0x7f4e5813b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 >> v1:172.21.15.50:6789/0 conn(0x7f4e5814be90 legacy=0x7f4e5812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 1 monclient: found mon.b 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5813aa90 con 0x7f4e580b4d60 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 10 monclient: _renew_subs 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5813ad70 con 0x7f4e580b4d60 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 10 monclient: _renew_subs 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e58145330 con 0x7f4e580b4d60 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.088+0000 7f4e5dd1c640 1 librados: init done 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.089+0000 7f4e5dd1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.089+0000 7f4e5dd1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.089+0000 7f4e5dd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:38.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.089+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e581401f0 con 0x7f4e580b4d60 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.374+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e4c004a90 con 0x7f4e580b4d60 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.374+0000 7f4e3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.374+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4e4c004da0 con 0x7f4e580b4d60 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.374+0000 7f4e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.374+0000 7f4e3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.374+0000 7f4e3ffff640 10 monclient: dump: 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:38.375 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.376 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:38.376 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:38.376 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:38.376 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:38.376 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:38.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.374+0000 7f4e55ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:38.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.375+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f4e4c01ad70 con 0x7f4e580b4d60 2024-03-25T19:43:38.377 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.376+0000 7f4e577fe640 10 monclient: get_auth_request con 0x7f4e4406e180 auth_method 0 2024-03-25T19:43:38.377 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.376+0000 7f4e3ffff640 10 client.4336.objecter ms_handle_connect 0x7f4e4406e180 2024-03-25T19:43:38.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.379+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f4e4c054900 con 0x7f4e580b4d60 2024-03-25T19:43:38.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.379+0000 7f4e3ffff640 10 client.4336.objecter ms_dispatch 0x7f4e58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:38.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.379+0000 7f4e3ffff640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:38.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.379+0000 7f4e3ffff640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:38.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.379+0000 7f4e3ffff640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T19:43:38.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.382+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e4c050410 con 0x7f4e580b4d60 2024-03-25T19:43:38.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.382+0000 7f4e3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.382+0000 7f4e3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:38.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.631+0000 7f4e5dd1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.631+0000 7f4e5dd1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.631+0000 7f4e5dd1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:43:38.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.631+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e580afbd0 con 0x7f4e580b4d60 2024-03-25T19:43:38.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.659+0000 7f4e3ffff640 1 -- 172.21.15.50:0/3903733737 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f4e4c052340 con 0x7f4e580b4d60 2024-03-25T19:43:38.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.659+0000 7f4e3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.659+0000 7f4e3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:38.660 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:38.660 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:38.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 10 librados: watch_flush enter 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 10 librados: watch_flush exit 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 10 client.4336.objecter successfully canceled tick 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 >> v1:172.21.15.76:6800/2572556811 conn(0x7f4e4406e180 legacy=0x7f4e44070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 10 monclient: shutdown 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 >> v1:172.21.15.76:6789/0 conn(0x7f4e580b4d60 legacy=0x7f4e5812e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 shutdown_connections 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 >> 172.21.15.50:0/3903733737 conn(0x7f4e580617a0 msgr2=0x7f4e5813c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 shutdown_connections 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.661+0000 7f4e5dd1c640 1 -- 172.21.15.50:0/3903733737 wait complete. 2024-03-25T19:43:38.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:38.662+0000 7f4e5dd1c640 1 librados: shutdown 2024-03-25T19:43:38.675 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:45.481 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:45.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.596+0000 7f4b250e0640 10 monclient: get_monmap_and_config 2024-03-25T19:43:45.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.596+0000 7f4b250e0640 10 monclient: build_initial_monmap 2024-03-25T19:43:45.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.596+0000 7f4b250e0640 10 monclient: monmap: 2024-03-25T19:43:45.596 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:45.596 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:43:45.597769+0000 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:43:45.597769+0000 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:43:45.597 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:45.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.597+0000 7f4b250e0640 1 Processor -- start 2024-03-25T19:43:45.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.597+0000 7f4b250e0640 1 -- start start 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.597+0000 7f4b250e0640 10 monclient: init 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 10 monclient(hunting): picked mon.noname-b con 0x7f4b200a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4b2013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4b201430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b20142520 con 0x7f4b2013f710 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200b28a0 con 0x7f4b201430c0 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200b2a30 con 0x7f4b200a98f0 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.598+0000 7f4b250e0640 10 monclient(hunting): authenticate will time out at 5000.862403s 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.599+0000 7f4b1ed9d640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f4b200a98f0 0x7f4b200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:39022/0 (socket says 172.21.15.50:39022) 2024-03-25T19:43:45.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.599+0000 7f4b1ed9d640 1 -- 172.21.15.50:0/2611070348 learned_addr learned my addr 172.21.15.50:0/2611070348 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.613+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137265484 0 0) 0x7f4b200b28a0 con 0x7f4b201430c0 2024-03-25T19:43:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.613+0000 7f4b1dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.613+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.613+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.613+0000 7f4b1dd9b640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:43:45.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.613+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b08003520 con 0x7f4b201430c0 2024-03-25T19:43:45.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.616+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170658596 0 0) 0x7f4b200b2a30 con 0x7f4b200a98f0 2024-03-25T19:43:45.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.616+0000 7f4b1dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.616+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.616+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.616+0000 7f4b1dd9b640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:43:45.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.616+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b200b28a0 con 0x7f4b200a98f0 2024-03-25T19:43:45.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.628+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318462169 0 0) 0x7f4b20142520 con 0x7f4b2013f710 2024-03-25T19:43:45.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.628+0000 7f4b1dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.628+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.628+0000 7f4b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.628+0000 7f4b1dd9b640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:43:45.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.628+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b200b2a30 con 0x7f4b2013f710 2024-03-25T19:43:45.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.630+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4176345260 0 0) 0x7f4b200b28a0 con 0x7f4b200a98f0 2024-03-25T19:43:45.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.630+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 >> v1:172.21.15.76:6789/0 conn(0x7f4b201430c0 legacy=0x7f4b2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.630+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 >> v1:172.21.15.50:6789/0 conn(0x7f4b2013f710 legacy=0x7f4b201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.630+0000 7f4b1dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.630+0000 7f4b1dd9b640 1 monclient: found mon.noname-b 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.630+0000 7f4b1dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b200b44c0 con 0x7f4b200a98f0 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4b10001670 con 0x7f4b200a98f0 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b1dd9b640 10 monclient: dump: 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.631 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b250e0640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:43:45.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.631+0000 7f4b250e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:45.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b100017d0 con 0x7f4b200a98f0 2024-03-25T19:43:45.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b1dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:45.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b1dd9b640 1 -- 172.21.15.50:0/2611070348 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4b10001be0 con 0x7f4b200a98f0 2024-03-25T19:43:45.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b250e0640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:45.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b250e0640 4 set_mon_vals no callback set 2024-03-25T19:43:45.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b1dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b1dd9b640 10 monclient: dump: 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:45.652 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:45.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b250e0640 10 monclient: shutdown 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b250e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b250e0640 1 -- 172.21.15.50:0/2611070348 >> v1:172.21.15.50:6790/0 conn(0x7f4b200a98f0 legacy=0x7f4b200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b250e0640 1 -- 172.21.15.50:0/2611070348 shutdown_connections 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.652+0000 7f4b250e0640 1 -- 172.21.15.50:0/2611070348 >> 172.21.15.50:0/2611070348 conn(0x7f4b200b7ea0 msgr2=0x7f4b200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.653+0000 7f4b250e0640 1 -- 172.21.15.50:0/2611070348 shutdown_connections 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.653+0000 7f4b250e0640 1 -- 172.21.15.50:0/2611070348 wait complete. 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.653+0000 7f4b250e0640 10 monclient: build_initial_monmap 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.653+0000 7f4b250e0640 10 monclient: monmap: 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:43:45.654 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.654+0000 7f4b250e0640 1 librados: starting msgr at 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.654+0000 7f4b250e0640 1 librados: starting objecter 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.654+0000 7f4b250e0640 1 Processor -- start 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.654+0000 7f4b250e0640 1 -- start start 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.654+0000 7f4b250e0640 1 librados: setting wanted keys 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.654+0000 7f4b250e0640 1 librados: calling monclient init 2024-03-25T19:43:45.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.654+0000 7f4b250e0640 10 monclient: init 2024-03-25T19:43:45.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:45.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:45.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 10 monclient(hunting): picked mon.noname-a con 0x7f4b200a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:45.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 10 monclient(hunting): picked mon.noname-b con 0x7f4b2013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:45.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4b201430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:45.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200abc80 con 0x7f4b200a98f0 2024-03-25T19:43:45.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200abf60 con 0x7f4b2013f710 2024-03-25T19:43:45.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b200ac240 con 0x7f4b201430c0 2024-03-25T19:43:45.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:45.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.655+0000 7f4b250e0640 10 monclient(hunting): authenticate will time out at 5000.918883s 2024-03-25T19:43:45.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.659+0000 7f4b1f59e640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f4b201430c0 0x7f4b201e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:39062/0 (socket says 172.21.15.50:39062) 2024-03-25T19:43:45.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.659+0000 7f4b1f59e640 1 -- 172.21.15.50:0/87629367 learned_addr learned my addr 172.21.15.50:0/87629367 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:45.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.666+0000 7f4b077fe640 10 client.?.objecter ms_handle_connect 0x7f4b201430c0 2024-03-25T19:43:45.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.666+0000 7f4b077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:45.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.668+0000 7f4b077fe640 10 client.?.objecter ms_handle_connect 0x7f4b200a98f0 2024-03-25T19:43:45.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.668+0000 7f4b077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:45.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.670+0000 7f4b077fe640 10 client.?.objecter ms_handle_connect 0x7f4b2013f710 2024-03-25T19:43:45.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.670+0000 7f4b077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:45.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.677+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845341439 0 0) 0x7f4b200ac240 con 0x7f4b201430c0 2024-03-25T19:43:45.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.677+0000 7f4b077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.677+0000 7f4b077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.677+0000 7f4b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.677+0000 7f4b077fe640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:43:45.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.677+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b0c004eb0 con 0x7f4b201430c0 2024-03-25T19:43:45.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.680+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114469165 0 0) 0x7f4b200abf60 con 0x7f4b2013f710 2024-03-25T19:43:45.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.680+0000 7f4b077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.680+0000 7f4b077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.680+0000 7f4b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.680+0000 7f4b077fe640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:43:45.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.680+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b200ac240 con 0x7f4b2013f710 2024-03-25T19:43:45.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.685+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263767714 0 0) 0x7f4b200abc80 con 0x7f4b200a98f0 2024-03-25T19:43:45.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.685+0000 7f4b077fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.685+0000 7f4b077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.685+0000 7f4b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.685+0000 7f4b077fe640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:43:45.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.685+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b200abf60 con 0x7f4b200a98f0 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438619724 0 0) 0x7f4b200ac240 con 0x7f4b2013f710 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b200abc80 con 0x7f4b2013f710 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4b14001870 con 0x7f4b2013f710 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 10 monclient(hunting): dump: 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.701 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 713601895 0 0) 0x7f4b200abf60 con 0x7f4b200a98f0 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b200ac240 con 0x7f4b200a98f0 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4b10001ae0 con 0x7f4b200a98f0 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.701+0000 7f4b077fe640 10 monclient(hunting): dump: 2024-03-25T19:43:45.702 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:45.703 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:45.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.707+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573662913 0 0) 0x7f4b0c004eb0 con 0x7f4b201430c0 2024-03-25T19:43:45.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.707+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b200abf60 con 0x7f4b201430c0 2024-03-25T19:43:45.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.707+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4b18001c30 con 0x7f4b201430c0 2024-03-25T19:43:45.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.707+0000 7f4b077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.707+0000 7f4b077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:45.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.707+0000 7f4b077fe640 10 monclient(hunting): dump: 2024-03-25T19:43:45.706 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:45.707 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1651034236 0 0) 0x7f4b200ac240 con 0x7f4b200a98f0 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 >> v1:172.21.15.50:6790/0 conn(0x7f4b201430c0 legacy=0x7f4b201e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 >> v1:172.21.15.76:6789/0 conn(0x7f4b2013f710 legacy=0x7f4b201365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 1 monclient: found mon.a 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b20138520 con 0x7f4b200a98f0 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:45.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 5 monclient: authenticate success, global_id 4359 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 10 monclient: _renew_subs 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b20137f50 con 0x7f4b200a98f0 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 10 monclient: _renew_subs 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b20138e20 con 0x7f4b200a98f0 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.714+0000 7f4b250e0640 1 librados: init done 2024-03-25T19:43:45.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.715+0000 7f4b250e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:45.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.715+0000 7f4b250e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:45.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.715+0000 7f4b250e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:45.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:45.715+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b200b3ef0 con 0x7f4b200a98f0 2024-03-25T19:43:46.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.039+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b10001210 con 0x7f4b200a98f0 2024-03-25T19:43:46.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.039+0000 7f4b077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:46.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.039+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4b10004c30 con 0x7f4b200a98f0 2024-03-25T19:43:46.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.039+0000 7f4b077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.039+0000 7f4b077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:46.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.039+0000 7f4b077fe640 10 monclient: dump: 2024-03-25T19:43:46.039 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:46.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.039+0000 7f4b1d59a640 4 set_mon_vals no callback set 2024-03-25T19:43:46.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.040+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f4b1001ac00 con 0x7f4b200a98f0 2024-03-25T19:43:46.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.040+0000 7f4b1e59c640 10 monclient: get_auth_request con 0x7f4b0c06e110 auth_method 0 2024-03-25T19:43:46.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.041+0000 7f4b077fe640 10 client.4359.objecter ms_handle_connect 0x7f4b0c06e110 2024-03-25T19:43:46.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f4b10054870 con 0x7f4b200a98f0 2024-03-25T19:43:46.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 10 client.4359.objecter ms_dispatch 0x7f4b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:46.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:46.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:46.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-25T19:43:46.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b10050380 con 0x7f4b200a98f0 2024-03-25T19:43:46.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.048+0000 7f4b077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:46.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.304+0000 7f4b250e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.304+0000 7f4b250e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.304+0000 7f4b250e0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:43:46.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.304+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b201382a0 con 0x7f4b200a98f0 2024-03-25T19:43:46.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.329+0000 7f4b077fe640 1 -- 172.21.15.50:0/87629367 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f4b100522b0 con 0x7f4b200a98f0 2024-03-25T19:43:46.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.329+0000 7f4b077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.329+0000 7f4b077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:46.329 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:46.329 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:46.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 10 librados: watch_flush enter 2024-03-25T19:43:46.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 10 librados: watch_flush exit 2024-03-25T19:43:46.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 10 client.4359.objecter successfully canceled tick 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 >> v1:172.21.15.76:6800/2572556811 conn(0x7f4b0c06e110 legacy=0x7f4b0c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 10 monclient: shutdown 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 >> v1:172.21.15.50:6789/0 conn(0x7f4b200a98f0 legacy=0x7f4b200ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 shutdown_connections 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 >> 172.21.15.50:0/87629367 conn(0x7f4b200b7ea0 msgr2=0x7f4b20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.332+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 shutdown_connections 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.333+0000 7f4b250e0640 1 -- 172.21.15.50:0/87629367 wait complete. 2024-03-25T19:43:46.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:46.333+0000 7f4b250e0640 1 librados: shutdown 2024-03-25T19:43:46.344 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:53.151 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:53.255 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.256+0000 7f83d1b65640 10 monclient: get_monmap_and_config 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.256+0000 7f83d1b65640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.256+0000 7f83d1b65640 10 monclient: monmap: 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:43:53.257274+0000 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:43:53.257274+0000 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:43:53.256 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:43:53.257 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.257 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.257+0000 7f83d1b65640 1 Processor -- start 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.257+0000 7f83d1b65640 1 -- start start 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.257+0000 7f83d1b65640 10 monclient: init 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.257+0000 7f83d1b65640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.257+0000 7f83d1b65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 10 monclient(hunting): picked mon.noname-b con 0x7f83cc0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 10 monclient(hunting): picked mon.noname-a con 0x7f83cc13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 10 monclient(hunting): picked mon.noname-c con 0x7f83cc1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc142520 con 0x7f83cc13f710 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc0b28a0 con 0x7f83cc1430c0 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc0b2a30 con 0x7f83cc0a98f0 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.258+0000 7f83d1b65640 10 monclient(hunting): authenticate will time out at 5008.521803s 2024-03-25T19:43:53.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.265+0000 7f83cbfff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f83cc1430c0 0x7f83cc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:37876/0 (socket says 172.21.15.50:37876) 2024-03-25T19:43:53.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.265+0000 7f83cbfff640 1 -- 172.21.15.50:0/1784780998 learned_addr learned my addr 172.21.15.50:0/1784780998 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:53.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.278+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193516168 0 0) 0x7f83cc0b28a0 con 0x7f83cc1430c0 2024-03-25T19:43:53.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.278+0000 7f83ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.278+0000 7f83ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.278+0000 7f83ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.278+0000 7f83ca7fc640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:43:53.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.278+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b4003520 con 0x7f83cc1430c0 2024-03-25T19:43:53.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.291+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185864190 0 0) 0x7f83cc0b2a30 con 0x7f83cc0a98f0 2024-03-25T19:43:53.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.291+0000 7f83ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.291+0000 7f83ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.291+0000 7f83ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.291+0000 7f83ca7fc640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:43:53.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.291+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83cc0b28a0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2141761036 0 0) 0x7f83b4003520 con 0x7f83cc1430c0 2024-03-25T19:43:53.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 >> v1:172.21.15.50:6790/0 conn(0x7f83cc0a98f0 legacy=0x7f83cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 >> v1:172.21.15.50:6789/0 conn(0x7f83cc13f710 legacy=0x7f83cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 1 monclient: found mon.noname-c 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc0b44c0 con 0x7f83cc1430c0 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f83c4001a10 con 0x7f83cc1430c0 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83ca7fc640 10 monclient: dump: 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:53.294 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83d1b65640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:43:53.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.293+0000 7f83d1b65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:53.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83c4001b70 con 0x7f83cc1430c0 2024-03-25T19:43:53.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:53.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83ca7fc640 1 -- 172.21.15.50:0/1784780998 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f83c4001f80 con 0x7f83cc1430c0 2024-03-25T19:43:53.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83ca7fc640 10 monclient: dump: 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83d1b65640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:53.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.313+0000 7f83d1b65640 4 set_mon_vals no callback set 2024-03-25T19:43:53.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.314+0000 7f83d1b65640 10 monclient: shutdown 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.314+0000 7f83d1b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.314+0000 7f83d1b65640 1 -- 172.21.15.50:0/1784780998 >> v1:172.21.15.76:6789/0 conn(0x7f83cc1430c0 legacy=0x7f83cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.314+0000 7f83d1b65640 1 -- 172.21.15.50:0/1784780998 shutdown_connections 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.314+0000 7f83d1b65640 1 -- 172.21.15.50:0/1784780998 >> 172.21.15.50:0/1784780998 conn(0x7f83cc0b7ea0 msgr2=0x7f83cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.314+0000 7f83d1b65640 1 -- 172.21.15.50:0/1784780998 shutdown_connections 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.314+0000 7f83d1b65640 1 -- 172.21.15.50:0/1784780998 wait complete. 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.315+0000 7f83d1b65640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.315+0000 7f83d1b65640 10 monclient: monmap: 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.316+0000 7f83d1b65640 1 librados: starting msgr at 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.316+0000 7f83d1b65640 1 librados: starting objecter 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.316+0000 7f83d1b65640 1 Processor -- start 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.316+0000 7f83d1b65640 1 -- start start 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.316+0000 7f83d1b65640 1 librados: setting wanted keys 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.316+0000 7f83d1b65640 1 librados: calling monclient init 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.316+0000 7f83d1b65640 10 monclient: init 2024-03-25T19:43:53.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:53.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 10 monclient(hunting): picked mon.noname-c con 0x7f83cc0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:43:53.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 10 monclient(hunting): picked mon.noname-a con 0x7f83cc13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:43:53.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 10 monclient(hunting): picked mon.noname-b con 0x7f83cc1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:43:53.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc136ec0 con 0x7f83cc13f710 2024-03-25T19:43:53.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc136c20 con 0x7f83cc1430c0 2024-03-25T19:43:53.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc137db0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.317+0000 7f83d1b65640 10 monclient(hunting): authenticate will time out at 5008.580983s 2024-03-25T19:43:53.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.329+0000 7f83caffd640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f83cc13f710 0x7f83cc0b0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59906/0 (socket says 172.21.15.50:59906) 2024-03-25T19:43:53.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.329+0000 7f83caffd640 1 -- 172.21.15.50:0/3237827713 learned_addr learned my addr 172.21.15.50:0/3237827713 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:43:53.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.332+0000 7f83b3fff640 10 client.?.objecter ms_handle_connect 0x7f83cc1430c0 2024-03-25T19:43:53.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.332+0000 7f83b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.333+0000 7f83b3fff640 10 client.?.objecter ms_handle_connect 0x7f83cc13f710 2024-03-25T19:43:53.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.333+0000 7f83b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.339+0000 7f83b3fff640 10 client.?.objecter ms_handle_connect 0x7f83cc0a98f0 2024-03-25T19:43:53.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.339+0000 7f83b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.340+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691362554 0 0) 0x7f83cc136ec0 con 0x7f83cc13f710 2024-03-25T19:43:53.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.340+0000 7f83b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.340+0000 7f83b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.340+0000 7f83b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.340+0000 7f83b3fff640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:43:53.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.340+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b8004eb0 con 0x7f83cc13f710 2024-03-25T19:43:53.345 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.346+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193136349 0 0) 0x7f83cc136c20 con 0x7f83cc1430c0 2024-03-25T19:43:53.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.346+0000 7f83b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.346+0000 7f83b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.346+0000 7f83b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.346+0000 7f83b3fff640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:43:53.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.346+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83cc136ec0 con 0x7f83cc1430c0 2024-03-25T19:43:53.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.351+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754306189 0 0) 0x7f83cc137db0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.351+0000 7f83b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.351+0000 7f83b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.351+0000 7f83b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.351+0000 7f83b3fff640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:43:53.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.351+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83cc136c20 con 0x7f83cc0a98f0 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.360+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2592917419 0 0) 0x7f83b8004eb0 con 0x7f83cc13f710 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.361+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83cc137db0 con 0x7f83cc13f710 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.361+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f83c0001a30 con 0x7f83cc13f710 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.361+0000 7f83b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.361+0000 7f83b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.361+0000 7f83b3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:53.362 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.361+0000 7f83b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:53.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.362+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 687572604 0 0) 0x7f83cc136c20 con 0x7f83cc0a98f0 2024-03-25T19:43:53.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.362+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83b8004eb0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.362+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f83bc001930 con 0x7f83cc0a98f0 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.362+0000 7f83b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.362+0000 7f83b3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.362+0000 7f83b3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.367+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 27519069 0 0) 0x7f83cc136ec0 con 0x7f83cc1430c0 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.367+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83cc136c20 con 0x7f83cc1430c0 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.367+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f83c4001e80 con 0x7f83cc1430c0 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.367+0000 7f83b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.367+0000 7f83b3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.367+0000 7f83b3fff640 10 monclient(hunting): dump: 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:53.367 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:53.368 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:53.368 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3709981635 0 0) 0x7f83b8004eb0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 >> v1:172.21.15.76:6789/0 conn(0x7f83cc1430c0 legacy=0x7f83cc1e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 >> v1:172.21.15.50:6789/0 conn(0x7f83cc13f710 legacy=0x7f83cc0b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 1 monclient: found mon.c 2024-03-25T19:43:53.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc138b50 con 0x7f83cc0a98f0 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 5 monclient: authenticate success, global_id 4349 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 10 monclient: _renew_subs 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83cc139830 con 0x7f83cc0a98f0 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 10 monclient: _renew_subs 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83cc1369b0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.380+0000 7f83d1b65640 1 librados: init done 2024-03-25T19:43:53.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.381+0000 7f83d1b65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.381+0000 7f83d1b65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.381+0000 7f83d1b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:53.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.381+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83cc139a20 con 0x7f83cc0a98f0 2024-03-25T19:43:53.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.697+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83bc003610 con 0x7f83cc0a98f0 2024-03-25T19:43:53.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.697+0000 7f83b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:53.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.697+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f83bc001d80 con 0x7f83cc0a98f0 2024-03-25T19:43:53.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.697+0000 7f83b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.697+0000 7f83b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:53.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.697+0000 7f83b3fff640 10 monclient: dump: 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:43:53.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.697+0000 7f83c9ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:53.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.698+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f83bc019670 con 0x7f83cc0a98f0 2024-03-25T19:43:53.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.699+0000 7f83caffd640 10 monclient: get_auth_request con 0x7f83b806e0b0 auth_method 0 2024-03-25T19:43:53.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.699+0000 7f83b3fff640 10 client.4349.objecter ms_handle_connect 0x7f83b806e0b0 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f83bc053360 con 0x7f83cc0a98f0 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 10 client.4349.objecter ms_dispatch 0x7f83cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83bc04eca0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.705+0000 7f83b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:53.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.969+0000 7f83d1b65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.969+0000 7f83d1b65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.969+0000 7f83d1b65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:43:53.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.969+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83cc0627b0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.995+0000 7f83b3fff640 1 -- 172.21.15.50:0/3237827713 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f83bc050da0 con 0x7f83cc0a98f0 2024-03-25T19:43:53.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.995+0000 7f83b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.995+0000 7f83b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:53.995 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:43:53.995 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.997+0000 7f83d1b65640 10 librados: watch_flush enter 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.997+0000 7f83d1b65640 10 librados: watch_flush exit 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.997+0000 7f83d1b65640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.997+0000 7f83d1b65640 10 client.4349.objecter successfully canceled tick 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 >> v1:172.21.15.76:6800/2572556811 conn(0x7f83b806e0b0 legacy=0x7f83b8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 10 monclient: shutdown 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 >> v1:172.21.15.50:6790/0 conn(0x7f83cc0a98f0 legacy=0x7f83cc1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 shutdown_connections 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 >> 172.21.15.50:0/3237827713 conn(0x7f83cc0b7ea0 msgr2=0x7f83cc13ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 shutdown_connections 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 1 -- 172.21.15.50:0/3237827713 wait complete. 2024-03-25T19:43:53.998 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:43:53.998+0000 7f83d1b65640 1 librados: shutdown 2024-03-25T19:43:54.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:54.414 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:14.919 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.462 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:14.932 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.465 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:14.971 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.478 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:14.949 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.489 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:14.995 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.513 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:14.984 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.522 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:15.028 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.530 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:15.036 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:54.568 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:15.074 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.815 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.931+0000 7f526a406640 10 monclient: get_monmap_and_config 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.931+0000 7f526a406640 10 monclient: build_initial_monmap 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.931+0000 7f526a406640 10 monclient: monmap: 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:00.932669+0000 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:00.932669+0000 2024-03-25T19:44:00.931 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:00.932 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:00.932 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:00.932 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:00.932 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:00.932 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.932+0000 7f526a406640 1 Processor -- start 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.932+0000 7f526a406640 1 -- start start 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.932+0000 7f526a406640 10 monclient: init 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 10 monclient(hunting): picked mon.noname-c con 0x7f52640a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 10 monclient(hunting): picked mon.noname-a con 0x7f526413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 10 monclient(hunting): picked mon.noname-b con 0x7f52641430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5264142520 con 0x7f526413f710 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52640b28a0 con 0x7f52640a98f0 2024-03-25T19:44:00.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52640b2a30 con 0x7f52641430c0 2024-03-25T19:44:00.934 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:00.934 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.933+0000 7f526a406640 10 monclient(hunting): authenticate will time out at 5016.197516s 2024-03-25T19:44:00.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.941+0000 7f5263fff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f52640a98f0 0x7f52640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:51712/0 (socket says 172.21.15.50:51712) 2024-03-25T19:44:00.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.941+0000 7f5263fff640 1 -- 172.21.15.50:0/2283705853 learned_addr learned my addr 172.21.15.50:0/2283705853 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:00.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.957+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263897445 0 0) 0x7f52640b28a0 con 0x7f52640a98f0 2024-03-25T19:44:00.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.957+0000 7f5262ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.957+0000 7f5262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.958+0000 7f5262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.958+0000 7f5262ffd640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:44:00.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.958+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f524c003520 con 0x7f52640a98f0 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758625648 0 0) 0x7f5264142520 con 0x7f526413f710 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52640b28a0 con 0x7f526413f710 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780810897 0 0) 0x7f52640b2a30 con 0x7f52641430c0 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.969+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5264142520 con 0x7f52641430c0 2024-03-25T19:44:00.977 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.977+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 328306754 0 0) 0x7f524c003520 con 0x7f52640a98f0 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.977+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 >> v1:172.21.15.50:6790/0 conn(0x7f52641430c0 legacy=0x7f526414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 >> v1:172.21.15.50:6789/0 conn(0x7f526413f710 legacy=0x7f52641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 1 monclient: found mon.noname-c 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52640b44c0 con 0x7f52640a98f0 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5254001690 con 0x7f52640a98f0 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f5262ffd640 10 monclient: dump: 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f526a406640 5 monclient: authenticate success, global_id 4351 2024-03-25T19:44:00.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.978+0000 7f526a406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:00.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52540017f0 con 0x7f52640a98f0 2024-03-25T19:44:00.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f5262ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f5262ffd640 1 -- 172.21.15.50:0/2283705853 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5254001c00 con 0x7f52640a98f0 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f526a406640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f526a406640 4 set_mon_vals no callback set 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f5262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f5262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.986+0000 7f5262ffd640 10 monclient: dump: 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:00.986 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:00.987 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.987+0000 7f526a406640 10 monclient: shutdown 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.987+0000 7f526a406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.987+0000 7f526a406640 1 -- 172.21.15.50:0/2283705853 >> v1:172.21.15.76:6789/0 conn(0x7f52640a98f0 legacy=0x7f52640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.987+0000 7f526a406640 1 -- 172.21.15.50:0/2283705853 shutdown_connections 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.987+0000 7f526a406640 1 -- 172.21.15.50:0/2283705853 >> 172.21.15.50:0/2283705853 conn(0x7f52640b7ea0 msgr2=0x7f52640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.987+0000 7f526a406640 1 -- 172.21.15.50:0/2283705853 shutdown_connections 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.987+0000 7f526a406640 1 -- 172.21.15.50:0/2283705853 wait complete. 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 10 monclient: build_initial_monmap 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 10 monclient: monmap: 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 1 librados: starting msgr at 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 1 librados: starting objecter 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 1 Processor -- start 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 1 -- start start 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 1 librados: setting wanted keys 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 1 librados: calling monclient init 2024-03-25T19:44:00.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.988+0000 7f526a406640 10 monclient: init 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 10 monclient(hunting): picked mon.noname-c con 0x7f52640a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 10 monclient(hunting): picked mon.noname-b con 0x7f526413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 10 monclient(hunting): picked mon.noname-a con 0x7f52641430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526406ff40 con 0x7f52641430c0 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5264070220 con 0x7f526413f710 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5264070500 con 0x7f52640a98f0 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.989+0000 7f526a406640 10 monclient(hunting): authenticate will time out at 5016.253653s 2024-03-25T19:44:00.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.991+0000 7f5263fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f52640a98f0 0x7f526406e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53598/0 (socket says 172.21.15.50:53598) 2024-03-25T19:44:00.992 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.991+0000 7f5263fff640 1 -- 172.21.15.50:0/2885224911 learned_addr learned my addr 172.21.15.50:0/2885224911 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:00.992 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.992+0000 7f5260ff9640 10 client.?.objecter ms_handle_connect 0x7f52640a98f0 2024-03-25T19:44:00.992 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.992+0000 7f5260ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:00.992 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.992+0000 7f5260ff9640 10 client.?.objecter ms_handle_connect 0x7f526413f710 2024-03-25T19:44:00.992 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:00.992+0000 7f5260ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.055+0000 7f5260ff9640 10 client.?.objecter ms_handle_connect 0x7f52641430c0 2024-03-25T19:44:01.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.055+0000 7f5260ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.063+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784666746 0 0) 0x7f526406ff40 con 0x7f52641430c0 2024-03-25T19:44:01.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.063+0000 7f5260ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.063+0000 7f5260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.063+0000 7f5260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.063+0000 7f5260ff9640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:44:01.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.063+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5250004eb0 con 0x7f52641430c0 2024-03-25T19:44:01.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.084+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1775828407 0 0) 0x7f5250004eb0 con 0x7f52641430c0 2024-03-25T19:44:01.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.084+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f526406ff40 con 0x7f52641430c0 2024-03-25T19:44:01.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.084+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f525c001bd0 con 0x7f52641430c0 2024-03-25T19:44:01.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.084+0000 7f5260ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.084+0000 7f5260ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.084+0000 7f5260ff9640 10 monclient(hunting): dump: 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:01.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.084+0000 7f5260ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:01.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.092+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078804669 0 0) 0x7f5264070220 con 0x7f526413f710 2024-03-25T19:44:01.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.092+0000 7f5260ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.092+0000 7f5260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.092+0000 7f5260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.092+0000 7f5260ff9640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:44:01.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.092+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5250004eb0 con 0x7f526413f710 2024-03-25T19:44:01.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1951390898 0 0) 0x7f526406ff40 con 0x7f52641430c0 2024-03-25T19:44:01.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 >> v1:172.21.15.50:6790/0 conn(0x7f52640a98f0 legacy=0x7f526406e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 >> v1:172.21.15.76:6789/0 conn(0x7f526413f710 legacy=0x7f5264071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:01.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 1 monclient: found mon.a 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526406f130 con 0x7f52641430c0 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f5260ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 10 monclient: _renew_subs 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526406dea0 con 0x7f52641430c0 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 10 monclient: _renew_subs 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f526414c0f0 con 0x7f52641430c0 2024-03-25T19:44:01.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.102+0000 7f526a406640 1 librados: init done 2024-03-25T19:44:01.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.103+0000 7f526a406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.103+0000 7f526a406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.103+0000 7f526a406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:01.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.103+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526414c3f0 con 0x7f52641430c0 2024-03-25T19:44:01.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.368+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f525c0020a0 con 0x7f52641430c0 2024-03-25T19:44:01.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.368+0000 7f5260ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:01.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.368+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f525c003990 con 0x7f52641430c0 2024-03-25T19:44:01.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.369+0000 7f5260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.369+0000 7f5260ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:01.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.369+0000 7f5260ff9640 10 monclient: dump: 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:01.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.369+0000 7f52627fc640 4 set_mon_vals no callback set 2024-03-25T19:44:01.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.369+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f525c003bb0 con 0x7f52641430c0 2024-03-25T19:44:01.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.370+0000 7f5263fff640 10 monclient: get_auth_request con 0x7f525006dd40 auth_method 0 2024-03-25T19:44:01.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.370+0000 7f5260ff9640 10 client.4371.objecter ms_handle_connect 0x7f525006dd40 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f525c053710 con 0x7f52641430c0 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 10 client.4371.objecter ms_dispatch 0x7f5264000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f525c04f050 con 0x7f52641430c0 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.374+0000 7f5260ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:01.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.605+0000 7f526a406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:01.606 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.606+0000 7f526a406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:01.606 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.606+0000 7f526a406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:01.606 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.606+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52640627b0 con 0x7f52641430c0 2024-03-25T19:44:01.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.638+0000 7f5260ff9640 1 -- 172.21.15.50:0/2885224911 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f525c051150 con 0x7f52641430c0 2024-03-25T19:44:01.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.638+0000 7f5260ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:01.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.638+0000 7f5260ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:01.638 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:01.639 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 10 librados: watch_flush enter 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 10 librados: watch_flush exit 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 10 client.4371.objecter successfully canceled tick 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 >> v1:172.21.15.76:6800/2572556811 conn(0x7f525006dd40 legacy=0x7f5250070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 10 monclient: shutdown 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:01.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 >> v1:172.21.15.50:6789/0 conn(0x7f52641430c0 legacy=0x7f52641e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 shutdown_connections 2024-03-25T19:44:01.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 >> 172.21.15.50:0/2885224911 conn(0x7f52640b7ea0 msgr2=0x7f5264074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:01.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 shutdown_connections 2024-03-25T19:44:01.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 1 -- 172.21.15.50:0/2885224911 wait complete. 2024-03-25T19:44:01.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:01.641+0000 7f526a406640 1 librados: shutdown 2024-03-25T19:44:01.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:08.458 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:08.574 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.574+0000 7fc963274640 10 monclient: get_monmap_and_config 2024-03-25T19:44:08.574 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.574+0000 7fc963274640 10 monclient: build_initial_monmap 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.574+0000 7fc963274640 10 monclient: monmap: 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:08.575717+0000 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:08.575717+0000 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:08.575 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.575+0000 7fc963274640 1 Processor -- start 2024-03-25T19:44:08.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.575+0000 7fc963274640 1 -- start start 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.575+0000 7fc963274640 10 monclient: init 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 10 monclient(hunting): picked mon.noname-c con 0x7fc95c0a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 10 monclient(hunting): picked mon.noname-a con 0x7fc95c13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 10 monclient(hunting): picked mon.noname-b con 0x7fc95c1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c142520 con 0x7fc95c13f710 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c0b28a0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c0b2a30 con 0x7fc95c1430c0 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:08.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.576+0000 7fc963274640 10 monclient(hunting): authenticate will time out at 5023.840155s 2024-03-25T19:44:08.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.585+0000 7fc961010640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc95c0a98f0 0x7fc95c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:48824/0 (socket says 172.21.15.50:48824) 2024-03-25T19:44:08.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.585+0000 7fc961010640 1 -- 172.21.15.50:0/1658826085 learned_addr learned my addr 172.21.15.50:0/1658826085 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:08.593 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.594+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259795905 0 0) 0x7fc95c0b28a0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.593 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.594+0000 7fc953fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.594+0000 7fc953fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.594+0000 7fc953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.594+0000 7fc953fff640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.594+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc948003520 con 0x7fc95c0a98f0 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.602+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678468580 0 0) 0x7fc95c142520 con 0x7fc95c13f710 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.602+0000 7fc953fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.602+0000 7fc953fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.602+0000 7fc953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.602+0000 7fc953fff640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.602+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c0b28a0 con 0x7fc95c13f710 2024-03-25T19:44:08.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.605+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396266452 0 0) 0x7fc95c0b2a30 con 0x7fc95c1430c0 2024-03-25T19:44:08.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.605+0000 7fc953fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.605+0000 7fc953fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.605+0000 7fc953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.605+0000 7fc953fff640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:44:08.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.605+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c142520 con 0x7fc95c1430c0 2024-03-25T19:44:08.606 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.606+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1518869417 0 0) 0x7fc948003520 con 0x7fc95c0a98f0 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.606+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 >> v1:172.21.15.50:6790/0 conn(0x7fc95c1430c0 legacy=0x7fc95c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 >> v1:172.21.15.50:6789/0 conn(0x7fc95c13f710 legacy=0x7fc95c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 1 monclient: found mon.noname-c 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0b44c0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc958001690 con 0x7fc95c0a98f0 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc953fff640 10 monclient: dump: 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc963274640 5 monclient: authenticate success, global_id 4357 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.607+0000 7fc963274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:08.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9580017f0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc953fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc953fff640 1 -- 172.21.15.50:0/1658826085 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc958001c00 con 0x7fc95c0a98f0 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc953fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc953fff640 10 monclient: dump: 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:08.629 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:08.630 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:08.630 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:08.630 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:08.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc963274640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:08.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.629+0000 7fc963274640 4 set_mon_vals no callback set 2024-03-25T19:44:08.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.630+0000 7fc963274640 10 monclient: shutdown 2024-03-25T19:44:08.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.630+0000 7fc963274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:08.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.630+0000 7fc963274640 1 -- 172.21.15.50:0/1658826085 >> v1:172.21.15.76:6789/0 conn(0x7fc95c0a98f0 legacy=0x7fc95c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.630+0000 7fc963274640 1 -- 172.21.15.50:0/1658826085 shutdown_connections 2024-03-25T19:44:08.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.630+0000 7fc963274640 1 -- 172.21.15.50:0/1658826085 >> 172.21.15.50:0/1658826085 conn(0x7fc95c0b7ea0 msgr2=0x7fc95c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.631+0000 7fc963274640 1 -- 172.21.15.50:0/1658826085 shutdown_connections 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.631+0000 7fc963274640 1 -- 172.21.15.50:0/1658826085 wait complete. 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.631+0000 7fc963274640 10 monclient: build_initial_monmap 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.631+0000 7fc963274640 10 monclient: monmap: 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:08.632 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:08.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.632+0000 7fc963274640 1 librados: starting msgr at 2024-03-25T19:44:08.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.632+0000 7fc963274640 1 librados: starting objecter 2024-03-25T19:44:08.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.632+0000 7fc963274640 1 Processor -- start 2024-03-25T19:44:08.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.632+0000 7fc963274640 1 -- start start 2024-03-25T19:44:08.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.632+0000 7fc963274640 1 librados: setting wanted keys 2024-03-25T19:44:08.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.632+0000 7fc963274640 1 librados: calling monclient init 2024-03-25T19:44:08.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.632+0000 7fc963274640 10 monclient: init 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 10 monclient(hunting): picked mon.noname-b con 0x7fc95c0a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 10 monclient(hunting): picked mon.noname-a con 0x7fc95c13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 10 monclient(hunting): picked mon.noname-c con 0x7fc95c1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c135590 con 0x7fc95c13f710 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c136e50 con 0x7fc95c0a98f0 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95c135070 con 0x7fc95c1430c0 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:08.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.633+0000 7fc963274640 10 monclient(hunting): authenticate will time out at 5023.897301s 2024-03-25T19:44:08.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.644+0000 7fc961010640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc95c0a98f0 0x7fc95c130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:48856/0 (socket says 172.21.15.50:48856) 2024-03-25T19:44:08.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.644+0000 7fc961010640 1 -- 172.21.15.50:0/4006984064 learned_addr learned my addr 172.21.15.50:0/4006984064 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:08.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.645+0000 7fc951ffb640 10 client.?.objecter ms_handle_connect 0x7fc95c0a98f0 2024-03-25T19:44:08.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.645+0000 7fc951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:08.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.650+0000 7fc951ffb640 10 client.?.objecter ms_handle_connect 0x7fc95c13f710 2024-03-25T19:44:08.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.650+0000 7fc951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.652+0000 7fc951ffb640 10 client.?.objecter ms_handle_connect 0x7fc95c1430c0 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.652+0000 7fc951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:08.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.661+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672528458 0 0) 0x7fc95c136e50 con 0x7fc95c0a98f0 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.661+0000 7fc951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.661+0000 7fc951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.661+0000 7fc951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.661+0000 7fc951ffb640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.661+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc940004eb0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.662+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144154462 0 0) 0x7fc95c135590 con 0x7fc95c13f710 2024-03-25T19:44:08.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.662+0000 7fc951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.662+0000 7fc951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.662+0000 7fc951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.662+0000 7fc951ffb640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:44:08.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.662+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c136e50 con 0x7fc95c13f710 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.669+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69695671 0 0) 0x7fc95c135070 con 0x7fc95c1430c0 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.669+0000 7fc951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.669+0000 7fc951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.669+0000 7fc951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.669+0000 7fc951ffb640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.669+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc95c135590 con 0x7fc95c1430c0 2024-03-25T19:44:08.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.674+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3768480088 0 0) 0x7fc940004eb0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.674+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc95c135070 con 0x7fc95c0a98f0 2024-03-25T19:44:08.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.674+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc958001bc0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.674+0000 7fc951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.674+0000 7fc951ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.674+0000 7fc951ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:08.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.674+0000 7fc951ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.683+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821124730 0 0) 0x7fc95c136e50 con 0x7fc95c13f710 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.684+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc940004eb0 con 0x7fc95c13f710 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.684+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc94c001970 con 0x7fc95c13f710 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.684+0000 7fc951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.684+0000 7fc951ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.684+0000 7fc951ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:08.683 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:08.684 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:08.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4250178992 0 0) 0x7fc95c135070 con 0x7fc95c0a98f0 2024-03-25T19:44:08.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 >> v1:172.21.15.50:6790/0 conn(0x7fc95c1430c0 legacy=0x7fc95c1e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 >> v1:172.21.15.50:6789/0 conn(0x7fc95c13f710 legacy=0x7fc95c1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 1 monclient: found mon.b 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c135d80 con 0x7fc95c0a98f0 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc963274640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc963274640 10 monclient: _renew_subs 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc963274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.686+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c131660 con 0x7fc95c0a98f0 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.687+0000 7fc963274640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.687+0000 7fc963274640 10 monclient: _renew_subs 2024-03-25T19:44:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.687+0000 7fc963274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:08.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.687+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c0b78a0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.687+0000 7fc963274640 1 librados: init done 2024-03-25T19:44:08.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.688+0000 7fc963274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:08.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.688+0000 7fc963274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:08.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.688+0000 7fc963274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:08.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.688+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c135f70 con 0x7fc95c0a98f0 2024-03-25T19:44:08.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9580011f0 con 0x7fc95c0a98f0 2024-03-25T19:44:08.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc958004b10 con 0x7fc95c0a98f0 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc951ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc951ffb640 10 monclient: dump: 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:08.961 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:08.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc9537fe640 4 set_mon_vals no callback set 2024-03-25T19:44:08.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.961+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fc95801ab20 con 0x7fc95c0a98f0 2024-03-25T19:44:08.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.962+0000 7fc96080f640 10 monclient: get_auth_request con 0x7fc94006de80 auth_method 0 2024-03-25T19:44:08.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.962+0000 7fc951ffb640 10 client.4360.objecter ms_handle_connect 0x7fc94006de80 2024-03-25T19:44:08.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fc958054840 con 0x7fc95c0a98f0 2024-03-25T19:44:08.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 10 client.4360.objecter ms_dispatch 0x7fc95c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:08.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:08.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:08.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T19:44:08.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc958050180 con 0x7fc95c0a98f0 2024-03-25T19:44:08.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:08.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:08.968+0000 7fc951ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:09.194 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.195+0000 7fc963274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.194 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.195+0000 7fc963274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.195 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.195+0000 7fc963274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:09.195 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.195+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc95c13d010 con 0x7fc95c0a98f0 2024-03-25T19:44:09.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.351+0000 7fc951ffb640 1 -- 172.21.15.50:0/4006984064 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fc958052280 con 0x7fc95c0a98f0 2024-03-25T19:44:09.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.351+0000 7fc951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.351+0000 7fc951ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:09.351 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:09.351 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:09.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 10 librados: watch_flush enter 2024-03-25T19:44:09.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 10 librados: watch_flush exit 2024-03-25T19:44:09.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T19:44:09.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 10 client.4360.objecter successfully canceled tick 2024-03-25T19:44:09.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 >> v1:172.21.15.76:6800/2572556811 conn(0x7fc94006de80 legacy=0x7fc940070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 10 monclient: shutdown 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 >> v1:172.21.15.76:6789/0 conn(0x7fc95c0a98f0 legacy=0x7fc95c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 shutdown_connections 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 >> 172.21.15.50:0/4006984064 conn(0x7fc95c0b7ea0 msgr2=0x7fc95c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 shutdown_connections 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 1 -- 172.21.15.50:0/4006984064 wait complete. 2024-03-25T19:44:09.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:09.353+0000 7fc963274640 1 librados: shutdown 2024-03-25T19:44:09.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:14.547 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:35.053 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.556 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:35.062 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.589 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:35.095 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.604 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:35.075 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.613 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:35.083 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.618 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:35.123 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.623 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:35.129 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.636 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:35.107 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:14.655 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:35.161 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.171 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:16.288 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.287+0000 7f1c05f6c640 10 monclient: get_monmap_and_config 2024-03-25T19:44:16.288 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.287+0000 7f1c05f6c640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.288 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.287+0000 7f1c05f6c640 10 monclient: monmap: 2024-03-25T19:44:16.288 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:16.288 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:16.288 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:16.289682+0000 2024-03-25T19:44:16.289 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:16.289682+0000 2024-03-25T19:44:16.289 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.289 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:16.289 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:16.289 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:16.289 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:16.289 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.288+0000 7f1c05f6c640 1 Processor -- start 2024-03-25T19:44:16.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.288+0000 7f1c05f6c640 1 -- start start 2024-03-25T19:44:16.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.288+0000 7f1c05f6c640 10 monclient: init 2024-03-25T19:44:16.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:16.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c000a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:16.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0013f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:16.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 10 monclient(hunting): picked mon.noname-c con 0x7f1c001430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:16.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c00142520 con 0x7f1c000a98f0 2024-03-25T19:44:16.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c000b28a0 con 0x7f1c001430c0 2024-03-25T19:44:16.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c000b2a30 con 0x7f1c0013f710 2024-03-25T19:44:16.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.289+0000 7f1c05f6c640 10 monclient(hunting): authenticate will time out at 5031.554312s 2024-03-25T19:44:16.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.300+0000 7f1bfffff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1c001430c0 0x7f1c0014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:53128/0 (socket says 172.21.15.50:53128) 2024-03-25T19:44:16.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.300+0000 7f1bfffff640 1 -- 172.21.15.50:0/2493284518 learned_addr learned my addr 172.21.15.50:0/2493284518 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:16.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.312+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334608386 0 0) 0x7f1c000b28a0 con 0x7f1c001430c0 2024-03-25T19:44:16.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.312+0000 7f1bfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.312+0000 7f1bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.312+0000 7f1bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.312+0000 7f1bfe7fc640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:44:16.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.312+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be8003520 con 0x7f1c001430c0 2024-03-25T19:44:16.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.320+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112508962 0 0) 0x7f1c00142520 con 0x7f1c000a98f0 2024-03-25T19:44:16.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.320+0000 7f1bfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.320+0000 7f1bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.320+0000 7f1bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.320+0000 7f1bfe7fc640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:44:16.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.320+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c000b28a0 con 0x7f1c000a98f0 2024-03-25T19:44:16.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.329+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247823578 0 0) 0x7f1c000b2a30 con 0x7f1c0013f710 2024-03-25T19:44:16.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.329+0000 7f1bfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.329+0000 7f1bfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.329+0000 7f1bfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.329+0000 7f1bfe7fc640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:44:16.330 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.329+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c00142520 con 0x7f1c0013f710 2024-03-25T19:44:16.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683748421 0 0) 0x7f1be8003520 con 0x7f1c001430c0 2024-03-25T19:44:16.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 >> v1:172.21.15.50:6790/0 conn(0x7f1c0013f710 legacy=0x7f1c001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 >> v1:172.21.15.50:6789/0 conn(0x7f1c000a98f0 legacy=0x7f1c000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:16.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 1 monclient: found mon.noname-c 2024-03-25T19:44:16.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:16.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c000b44c0 con 0x7f1c001430c0 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1bf8001a10 con 0x7f1c001430c0 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1c05f6c640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1c05f6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1bfe7fc640 10 monclient: dump: 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.339 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.340 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:16.340 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:16.340 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:16.340 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:16.340 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.337+0000 7f1c05f6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:16.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bf8001b70 con 0x7f1c001430c0 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1bfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1bfe7fc640 1 -- 172.21.15.50:0/2493284518 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1bf8001f80 con 0x7f1c001430c0 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1bfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1bfe7fc640 10 monclient: dump: 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:16.351 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1c05f6c640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.350+0000 7f1c05f6c640 4 set_mon_vals no callback set 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.351+0000 7f1c05f6c640 10 monclient: shutdown 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.351+0000 7f1c05f6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.351+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2493284518 >> v1:172.21.15.76:6789/0 conn(0x7f1c001430c0 legacy=0x7f1c0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.351+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2493284518 shutdown_connections 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.351+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2493284518 >> 172.21.15.50:0/2493284518 conn(0x7f1c000b7ea0 msgr2=0x7f1c000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.351+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2493284518 shutdown_connections 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.351+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2493284518 wait complete. 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 10 monclient: monmap: 2024-03-25T19:44:16.354 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 1 librados: starting msgr at 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 1 librados: starting objecter 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 1 Processor -- start 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 1 -- start start 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 1 librados: setting wanted keys 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 1 librados: calling monclient init 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.352+0000 7f1c05f6c640 10 monclient: init 2024-03-25T19:44:16.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c000a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 10 monclient(hunting): picked mon.noname-b con 0x7f1c0013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 10 monclient(hunting): picked mon.noname-c con 0x7f1c001430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c001369d0 con 0x7f1c000a98f0 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c00133a60 con 0x7f1c0013f710 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c000ab9e0 con 0x7f1c001430c0 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.353+0000 7f1c05f6c640 10 monclient(hunting): authenticate will time out at 5031.618188s 2024-03-25T19:44:16.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.362+0000 7f1bff7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1c000a98f0 0x7f1c00132c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47600/0 (socket says 172.21.15.50:47600) 2024-03-25T19:44:16.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.362+0000 7f1bff7fe640 1 -- 172.21.15.50:0/2701622199 learned_addr learned my addr 172.21.15.50:0/2701622199 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:16.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.371+0000 7f1be7fff640 10 client.?.objecter ms_handle_connect 0x7f1c000a98f0 2024-03-25T19:44:16.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.371+0000 7f1be7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.372+0000 7f1be7fff640 10 client.?.objecter ms_handle_connect 0x7f1c001430c0 2024-03-25T19:44:16.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.372+0000 7f1be7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.373+0000 7f1be7fff640 10 client.?.objecter ms_handle_connect 0x7f1c0013f710 2024-03-25T19:44:16.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.373+0000 7f1be7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.385+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141010991 0 0) 0x7f1c001369d0 con 0x7f1c000a98f0 2024-03-25T19:44:16.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.385+0000 7f1be7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.385+0000 7f1be7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.385+0000 7f1be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.385+0000 7f1be7fff640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:44:16.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.385+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bec004eb0 con 0x7f1c000a98f0 2024-03-25T19:44:16.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.389+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664854885 0 0) 0x7f1c00133a60 con 0x7f1c0013f710 2024-03-25T19:44:16.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.389+0000 7f1be7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.389+0000 7f1be7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.389+0000 7f1be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.389+0000 7f1be7fff640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:44:16.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.389+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c001369d0 con 0x7f1c0013f710 2024-03-25T19:44:16.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.391+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483442716 0 0) 0x7f1c000ab9e0 con 0x7f1c001430c0 2024-03-25T19:44:16.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.391+0000 7f1be7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.391+0000 7f1be7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.391+0000 7f1be7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.391+0000 7f1be7fff640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:44:16.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.391+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c00133a60 con 0x7f1c001430c0 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.400+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 80128428 0 0) 0x7f1c001369d0 con 0x7f1c0013f710 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.400+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c000ab9e0 con 0x7f1c0013f710 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.401+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1bf4001870 con 0x7f1c0013f710 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.401+0000 7f1be7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.401+0000 7f1be7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.401+0000 7f1be7fff640 10 monclient(hunting): dump: 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:16.401 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.401+0000 7f1be7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:16.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.412+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2079184906 0 0) 0x7f1c00133a60 con 0x7f1c001430c0 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.412+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c001369d0 con 0x7f1c001430c0 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.412+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1bf8001e80 con 0x7f1c001430c0 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.412+0000 7f1be7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.412+0000 7f1be7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.412+0000 7f1be7fff640 10 monclient(hunting): dump: 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:16.413 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.421+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2896345462 0 0) 0x7f1bec004eb0 con 0x7f1c000a98f0 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.421+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c00133a60 con 0x7f1c000a98f0 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.421+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1bf0001a30 con 0x7f1c000a98f0 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.421+0000 7f1be7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.421+0000 7f1be7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.421+0000 7f1be7fff640 10 monclient(hunting): dump: 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:16.422 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:16.423 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.422+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 883712815 0 0) 0x7f1c000ab9e0 con 0x7f1c0013f710 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 >> v1:172.21.15.50:6790/0 conn(0x7f1c001430c0 legacy=0x7f1c00133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 >> v1:172.21.15.50:6789/0 conn(0x7f1c000a98f0 legacy=0x7f1c00132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 1 monclient: found mon.b 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c000abcc0 con 0x7f1c0013f710 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1be7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 10 monclient: _renew_subs 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c000abfa0 con 0x7f1c0013f710 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:16.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 10 monclient: _renew_subs 2024-03-25T19:44:16.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:16.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c00137e40 con 0x7f1c0013f710 2024-03-25T19:44:16.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.423+0000 7f1c05f6c640 1 librados: init done 2024-03-25T19:44:16.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.424+0000 7f1c05f6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.424+0000 7f1c05f6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.424+0000 7f1c05f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:16.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.424+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c000627b0 con 0x7f1c0013f710 2024-03-25T19:44:16.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.702+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bf4003610 con 0x7f1c0013f710 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.702+0000 7f1be7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.702+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1bf4001d80 con 0x7f1c0013f710 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.702+0000 7f1be7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.702+0000 7f1be7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.702+0000 7f1be7fff640 10 monclient: dump: 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:16.703 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:16.704 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:16.704 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:16.704 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:16.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.702+0000 7f1bfdffb640 4 set_mon_vals no callback set 2024-03-25T19:44:16.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.703+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f1bf4019810 con 0x7f1c0013f710 2024-03-25T19:44:16.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.703+0000 7f1bff7fe640 10 monclient: get_auth_request con 0x7f1bec06e0f0 auth_method 0 2024-03-25T19:44:16.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.703+0000 7f1be7fff640 10 client.4366.objecter ms_handle_connect 0x7f1bec06e0f0 2024-03-25T19:44:16.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f1bf40534e0 con 0x7f1c0013f710 2024-03-25T19:44:16.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 10 client.4366.objecter ms_dispatch 0x7f1c00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:16.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:16.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:16.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:44:16.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bf404ee20 con 0x7f1c0013f710 2024-03-25T19:44:16.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.708+0000 7f1be7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:16.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.934+0000 7f1c05f6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.934+0000 7f1c05f6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.934+0000 7f1c05f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:16.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.934+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c000008d0 con 0x7f1c0013f710 2024-03-25T19:44:16.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.957+0000 7f1be7fff640 1 -- 172.21.15.50:0/2701622199 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f1bf4050f20 con 0x7f1c0013f710 2024-03-25T19:44:16.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.957+0000 7f1be7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.957+0000 7f1be7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:16.958 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:16.958 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:16.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.959+0000 7f1c05f6c640 10 librados: watch_flush enter 2024-03-25T19:44:16.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.959+0000 7f1c05f6c640 10 librados: watch_flush exit 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.959+0000 7f1c05f6c640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.959+0000 7f1c05f6c640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 >> v1:172.21.15.76:6800/2572556811 conn(0x7f1bec06e0f0 legacy=0x7f1bec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 10 monclient: shutdown 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 >> v1:172.21.15.76:6789/0 conn(0x7f1c0013f710 legacy=0x7f1c001362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 shutdown_connections 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 >> 172.21.15.50:0/2701622199 conn(0x7f1c000b7ea0 msgr2=0x7f1c00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 shutdown_connections 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 1 -- 172.21.15.50:0/2701622199 wait complete. 2024-03-25T19:44:16.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:16.960+0000 7f1c05f6c640 1 librados: shutdown 2024-03-25T19:44:16.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:23.780 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:23.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.896+0000 7f39267d6640 10 monclient: get_monmap_and_config 2024-03-25T19:44:23.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.896+0000 7f39267d6640 10 monclient: build_initial_monmap 2024-03-25T19:44:23.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.896+0000 7f39267d6640 10 monclient: monmap: 2024-03-25T19:44:23.897 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:23.898 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:23.898 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:23.897543+0000 2024-03-25T19:44:23.898 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:23.897543+0000 2024-03-25T19:44:23.898 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:23.898 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:23.898 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:23.898 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:23.899 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:23.899 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:23.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.897+0000 7f39267d6640 1 Processor -- start 2024-03-25T19:44:23.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.897+0000 7f39267d6640 1 -- start start 2024-03-25T19:44:23.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.897+0000 7f39267d6640 10 monclient: init 2024-03-25T19:44:23.901 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:23.901 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:23.901 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 10 monclient(hunting): picked mon.noname-b con 0x7f39200a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:23.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 10 monclient(hunting): picked mon.noname-c con 0x7f392013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:23.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 10 monclient(hunting): picked mon.noname-a con 0x7f39201430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:23.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3920142520 con 0x7f39201430c0 2024-03-25T19:44:23.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39200b28a0 con 0x7f392013f710 2024-03-25T19:44:23.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39200b2a30 con 0x7f39200a98f0 2024-03-25T19:44:23.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:23.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.898+0000 7f39267d6640 10 monclient(hunting): authenticate will time out at 5039.162263s 2024-03-25T19:44:23.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.910+0000 7f3924d73640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f39201430c0 0x7f392014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:47134/0 (socket says 172.21.15.50:47134) 2024-03-25T19:44:23.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.910+0000 7f3924d73640 1 -- 172.21.15.50:0/2621636741 learned_addr learned my addr 172.21.15.50:0/2621636741 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:23.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.920+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162862057 0 0) 0x7f3920142520 con 0x7f39201430c0 2024-03-25T19:44:23.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.920+0000 7f391effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.920+0000 7f391effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.920+0000 7f391effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.920+0000 7f391effd640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:44:23.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.920+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3908003520 con 0x7f39201430c0 2024-03-25T19:44:23.936 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2697732354 0 0) 0x7f3908003520 con 0x7f39201430c0 2024-03-25T19:44:23.936 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 >> v1:172.21.15.50:6790/0 conn(0x7f39200a98f0 legacy=0x7f39200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 >> v1:172.21.15.76:6789/0 conn(0x7f392013f710 legacy=0x7f39201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 1 monclient: found mon.noname-a 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39200b44c0 con 0x7f39201430c0 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 10 monclient: _finish_auth 0 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3918001a10 con 0x7f39201430c0 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:23.937 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.936+0000 7f391effd640 10 monclient: dump: 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.937+0000 7f39267d6640 5 monclient: authenticate success, global_id 4386 2024-03-25T19:44:23.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:23.937+0000 7f39267d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3918001b70 con 0x7f39201430c0 2024-03-25T19:44:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f391effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f391effd640 1 -- 172.21.15.50:0/2621636741 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3918001f80 con 0x7f39201430c0 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f391effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f391effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f391effd640 10 monclient: dump: 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:24.008 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:24.009 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:24.009 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f39267d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:24.009 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.008+0000 7f39267d6640 4 set_mon_vals no callback set 2024-03-25T19:44:24.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 10 monclient: shutdown 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 1 -- 172.21.15.50:0/2621636741 >> v1:172.21.15.50:6789/0 conn(0x7f39201430c0 legacy=0x7f392014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 1 -- 172.21.15.50:0/2621636741 shutdown_connections 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 1 -- 172.21.15.50:0/2621636741 >> 172.21.15.50:0/2621636741 conn(0x7f39200b7ea0 msgr2=0x7f39200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 1 -- 172.21.15.50:0/2621636741 shutdown_connections 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 1 -- 172.21.15.50:0/2621636741 wait complete. 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 10 monclient: build_initial_monmap 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.009+0000 7f39267d6640 10 monclient: monmap: 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:24.011 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 1 librados: starting msgr at 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 1 librados: starting objecter 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 1 Processor -- start 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 1 -- start start 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 1 librados: setting wanted keys 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 1 librados: calling monclient init 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 10 monclient: init 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.010+0000 7f39267d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 10 monclient(hunting): picked mon.noname-b con 0x7f39200a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 10 monclient(hunting): picked mon.noname-a con 0x7f392013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 10 monclient(hunting): picked mon.noname-c con 0x7f39201430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:24.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39200abac0 con 0x7f392013f710 2024-03-25T19:44:24.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39200abda0 con 0x7f39200a98f0 2024-03-25T19:44:24.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39201380e0 con 0x7f39201430c0 2024-03-25T19:44:24.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:24.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.011+0000 7f39267d6640 10 monclient(hunting): authenticate will time out at 5039.274799s 2024-03-25T19:44:24.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.016+0000 7f391ffff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f39200a98f0 0x7f39200ac580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:43674/0 (socket says 172.21.15.50:43674) 2024-03-25T19:44:24.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.016+0000 7f391ffff640 1 -- 172.21.15.50:0/107338393 learned_addr learned my addr 172.21.15.50:0/107338393 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:24.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.017+0000 7f391cff9640 10 client.?.objecter ms_handle_connect 0x7f39200a98f0 2024-03-25T19:44:24.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.017+0000 7f391cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.021+0000 7f391cff9640 10 client.?.objecter ms_handle_connect 0x7f39201430c0 2024-03-25T19:44:24.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.021+0000 7f391cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.025+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053466597 0 0) 0x7f39200abda0 con 0x7f39200a98f0 2024-03-25T19:44:24.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.025+0000 7f391cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.025+0000 7f391cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.025+0000 7f391cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.025+0000 7f391cff9640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:44:24.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.025+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f390c004eb0 con 0x7f39200a98f0 2024-03-25T19:44:24.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.027+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109946136 0 0) 0x7f39201380e0 con 0x7f39201430c0 2024-03-25T19:44:24.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.027+0000 7f391cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.027+0000 7f391cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.027+0000 7f391cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.027+0000 7f391cff9640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:44:24.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.027+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39200abda0 con 0x7f39201430c0 2024-03-25T19:44:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.035+0000 7f391cff9640 10 client.?.objecter ms_handle_connect 0x7f392013f710 2024-03-25T19:44:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.035+0000 7f391cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.045+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3325300654 0 0) 0x7f39200abda0 con 0x7f39201430c0 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.045+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39201380e0 con 0x7f39201430c0 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.045+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3918001e80 con 0x7f39201430c0 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.045+0000 7f391cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.045+0000 7f391cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.045+0000 7f391cff9640 10 monclient(hunting): dump: 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.045 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:24.046 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:24.046 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:24.046 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:24.046 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:24.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.045+0000 7f391cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:24.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.048+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821992273 0 0) 0x7f39200abac0 con 0x7f392013f710 2024-03-25T19:44:24.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.048+0000 7f391cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:24.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.048+0000 7f391cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:24.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.048+0000 7f391cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.048+0000 7f391cff9640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.048+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39200abda0 con 0x7f392013f710 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.049+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1850526437 0 0) 0x7f390c004eb0 con 0x7f39200a98f0 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.049+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39200abac0 con 0x7f39200a98f0 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.049+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3910001a30 con 0x7f39200a98f0 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.049+0000 7f391cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.049+0000 7f391cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.049+0000 7f391cff9640 10 monclient(hunting): dump: 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.049 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.050 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:24.050 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:24.050 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:24.050 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:24.050 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:24.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3762754596 0 0) 0x7f39201380e0 con 0x7f39201430c0 2024-03-25T19:44:24.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 >> v1:172.21.15.76:6789/0 conn(0x7f39200a98f0 legacy=0x7f39200ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 >> v1:172.21.15.50:6789/0 conn(0x7f392013f710 legacy=0x7f3920138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 1 monclient: found mon.c 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920137d90 con 0x7f39201430c0 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f391cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 5 monclient: authenticate success, global_id 4370 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 10 monclient: _renew_subs 2024-03-25T19:44:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3920139240 con 0x7f39201430c0 2024-03-25T19:44:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 10 monclient: _renew_subs 2024-03-25T19:44:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3920136830 con 0x7f39201430c0 2024-03-25T19:44:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.053+0000 7f39267d6640 1 librados: init done 2024-03-25T19:44:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.055+0000 7f39267d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.055+0000 7f39267d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.055+0000 7f39267d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:24.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.055+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39200627b0 con 0x7f39201430c0 2024-03-25T19:44:24.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.366+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3918002080 con 0x7f39201430c0 2024-03-25T19:44:24.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.366+0000 7f391cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:24.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.366+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3918004fb0 con 0x7f39201430c0 2024-03-25T19:44:24.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.366+0000 7f391cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.366+0000 7f391cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:24.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.366+0000 7f391cff9640 10 monclient: dump: 2024-03-25T19:44:24.368 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:24.368 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:24.368 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.368 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:24.368 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.369 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:24.369 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:24.369 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:24.369 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:24.369 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:24.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.366+0000 7f391e7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:24.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.367+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f391801af80 con 0x7f39201430c0 2024-03-25T19:44:24.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.368+0000 7f391ffff640 10 monclient: get_auth_request con 0x7f390c06dfb0 auth_method 0 2024-03-25T19:44:24.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.368+0000 7f391cff9640 10 client.4370.objecter ms_handle_connect 0x7f390c06dfb0 2024-03-25T19:44:24.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f3918054b60 con 0x7f39201430c0 2024-03-25T19:44:24.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 10 client.4370.objecter ms_dispatch 0x7f3920000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:24.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:24.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:24.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T19:44:24.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39180504a0 con 0x7f39201430c0 2024-03-25T19:44:24.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.373+0000 7f391cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:24.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.621+0000 7f39267d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.621+0000 7f39267d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.621+0000 7f39267d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:24.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.621+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39201393d0 con 0x7f39201430c0 2024-03-25T19:44:24.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.656+0000 7f391cff9640 1 -- 172.21.15.50:0/107338393 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f39180525a0 con 0x7f39201430c0 2024-03-25T19:44:24.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.656+0000 7f391cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.656+0000 7f391cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:24.656 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:24.656 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.658+0000 7f39267d6640 10 librados: watch_flush enter 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.658+0000 7f39267d6640 10 librados: watch_flush exit 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.658+0000 7f39267d6640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.658+0000 7f39267d6640 10 client.4370.objecter successfully canceled tick 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 >> v1:172.21.15.76:6800/2572556811 conn(0x7f390c06dfb0 legacy=0x7f390c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 10 monclient: shutdown 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 >> v1:172.21.15.50:6790/0 conn(0x7f39201430c0 legacy=0x7f39201e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 shutdown_connections 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 >> 172.21.15.50:0/107338393 conn(0x7f39200b7ea0 msgr2=0x7f3920074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 shutdown_connections 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 1 -- 172.21.15.50:0/107338393 wait complete. 2024-03-25T19:44:24.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:24.659+0000 7f39267d6640 1 librados: shutdown 2024-03-25T19:44:24.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:31.477 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.592+0000 7f716689d640 10 monclient: get_monmap_and_config 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.592+0000 7f716689d640 10 monclient: build_initial_monmap 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.592+0000 7f716689d640 10 monclient: monmap: 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:31.593649+0000 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:31.593649+0000 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:31.592 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:31.593 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:31.593 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:31.593 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:31.593 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:31.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.593+0000 7f716689d640 1 Processor -- start 2024-03-25T19:44:31.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.593+0000 7f716689d640 1 -- start start 2024-03-25T19:44:31.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.593+0000 7f716689d640 10 monclient: init 2024-03-25T19:44:31.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:31.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:31.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 10 monclient(hunting): picked mon.noname-b con 0x7f71600b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:31.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 10 monclient(hunting): picked mon.noname-a con 0x7f71600b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:31.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 10 monclient(hunting): picked mon.noname-c con 0x7f716014be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:31.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600af940 con 0x7f71600b1730 2024-03-25T19:44:31.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600b0a70 con 0x7f716014be90 2024-03-25T19:44:31.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600b0d50 con 0x7f71600b4d60 2024-03-25T19:44:31.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:31.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.594+0000 7f716689d640 10 monclient(hunting): authenticate will time out at 5046.858505s 2024-03-25T19:44:31.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.608+0000 7f715f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f71600b1730 0x7f71600b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41334/0 (socket says 172.21.15.50:41334) 2024-03-25T19:44:31.607 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.608+0000 7f715f7fe640 1 -- 172.21.15.50:0/2449101480 learned_addr learned my addr 172.21.15.50:0/2449101480 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:31.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.621+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905975145 0 0) 0x7f71600af940 con 0x7f71600b1730 2024-03-25T19:44:31.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.621+0000 7f715effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.621+0000 7f715effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.621+0000 7f715effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.621+0000 7f715effd640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:44:31.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.621+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7148003520 con 0x7f71600b1730 2024-03-25T19:44:31.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.632+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817174130 0 0) 0x7f71600b0d50 con 0x7f71600b4d60 2024-03-25T19:44:31.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.632+0000 7f715effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.632+0000 7f715effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.632+0000 7f715effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.632+0000 7f715effd640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:44:31.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.632+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71600af940 con 0x7f71600b4d60 2024-03-25T19:44:31.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.633+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906842427 0 0) 0x7f71600b0a70 con 0x7f716014be90 2024-03-25T19:44:31.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.633+0000 7f715effd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.633+0000 7f715effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.633+0000 7f715effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.633+0000 7f715effd640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:44:31.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.633+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71600b0d50 con 0x7f716014be90 2024-03-25T19:44:31.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540382486 0 0) 0x7f7148003520 con 0x7f71600b1730 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 >> v1:172.21.15.50:6790/0 conn(0x7f71600b4d60 legacy=0x7f716013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 >> v1:172.21.15.76:6789/0 conn(0x7f716014be90 legacy=0x7f716014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 1 monclient: found mon.noname-a 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71600b1030 con 0x7f71600b1730 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 10 monclient: _finish_auth 0 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7154001690 con 0x7f71600b1730 2024-03-25T19:44:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f715effd640 10 monclient: dump: 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f716689d640 5 monclient: authenticate success, global_id 4392 2024-03-25T19:44:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.636+0000 7f716689d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71540017f0 con 0x7f71600b1730 2024-03-25T19:44:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f715effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f715effd640 1 -- 172.21.15.50:0/2449101480 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7154001c00 con 0x7f71600b1730 2024-03-25T19:44:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f715effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f715effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f715effd640 10 monclient: dump: 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f716689d640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:31.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.644+0000 7f716689d640 4 set_mon_vals no callback set 2024-03-25T19:44:31.646 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 10 monclient: shutdown 2024-03-25T19:44:31.646 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 1 -- 172.21.15.50:0/2449101480 >> v1:172.21.15.50:6789/0 conn(0x7f71600b1730 legacy=0x7f71600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 1 -- 172.21.15.50:0/2449101480 shutdown_connections 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 1 -- 172.21.15.50:0/2449101480 >> 172.21.15.50:0/2449101480 conn(0x7f71600617a0 msgr2=0x7f7160061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 1 -- 172.21.15.50:0/2449101480 shutdown_connections 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 1 -- 172.21.15.50:0/2449101480 wait complete. 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 10 monclient: build_initial_monmap 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.645+0000 7f716689d640 10 monclient: monmap: 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.646+0000 7f716689d640 1 librados: starting msgr at 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.646+0000 7f716689d640 1 librados: starting objecter 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.646+0000 7f716689d640 1 Processor -- start 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.646+0000 7f716689d640 1 -- start start 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.646+0000 7f716689d640 1 librados: setting wanted keys 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.646+0000 7f716689d640 1 librados: calling monclient init 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.646+0000 7f716689d640 10 monclient: init 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 10 monclient(hunting): picked mon.noname-c con 0x7f71600b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 10 monclient(hunting): picked mon.noname-a con 0x7f71600b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 10 monclient(hunting): picked mon.noname-b con 0x7f716014be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:31.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7160144d40 con 0x7f71600b4d60 2024-03-25T19:44:31.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71601dbac0 con 0x7f716014be90 2024-03-25T19:44:31.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7160135d30 con 0x7f71600b1730 2024-03-25T19:44:31.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:31.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.647+0000 7f716689d640 10 monclient(hunting): authenticate will time out at 5046.911050s 2024-03-25T19:44:31.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.654+0000 7f715f7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f71600b4d60 0x7f7160144f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41382/0 (socket says 172.21.15.50:41382) 2024-03-25T19:44:31.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.654+0000 7f715f7fe640 1 -- 172.21.15.50:0/1228519950 learned_addr learned my addr 172.21.15.50:0/1228519950 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:31.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.655+0000 7f715cff9640 10 client.?.objecter ms_handle_connect 0x7f71600b4d60 2024-03-25T19:44:31.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.655+0000 7f715cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.664+0000 7f715cff9640 10 client.?.objecter ms_handle_connect 0x7f716014be90 2024-03-25T19:44:31.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.664+0000 7f715cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.669+0000 7f715cff9640 10 client.?.objecter ms_handle_connect 0x7f71600b1730 2024-03-25T19:44:31.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.669+0000 7f715cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.681+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647420117 0 0) 0x7f7160135d30 con 0x7f71600b1730 2024-03-25T19:44:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.681+0000 7f715cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.682+0000 7f715cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.682+0000 7f715cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.682+0000 7f715cff9640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:44:31.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.682+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c004eb0 con 0x7f71600b1730 2024-03-25T19:44:31.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.685+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641317241 0 0) 0x7f71601dbac0 con 0x7f716014be90 2024-03-25T19:44:31.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.685+0000 7f715cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.685+0000 7f715cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.685+0000 7f715cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.685+0000 7f715cff9640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:44:31.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.685+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7160135d30 con 0x7f716014be90 2024-03-25T19:44:31.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.695+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155552467 0 0) 0x7f7160144d40 con 0x7f71600b4d60 2024-03-25T19:44:31.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.695+0000 7f715cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.695+0000 7f715cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.695+0000 7f715cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.695+0000 7f715cff9640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:44:31.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.695+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71601dbac0 con 0x7f71600b4d60 2024-03-25T19:44:31.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.699+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1678310992 0 0) 0x7f7160135d30 con 0x7f716014be90 2024-03-25T19:44:31.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.699+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7160144d40 con 0x7f716014be90 2024-03-25T19:44:31.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.699+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7158001b50 con 0x7f716014be90 2024-03-25T19:44:31.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.699+0000 7f715cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.699+0000 7f715cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:31.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.699+0000 7f715cff9640 10 monclient(hunting): dump: 2024-03-25T19:44:31.699 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:31.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.699+0000 7f715cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.709+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126148020 0 0) 0x7f714c004eb0 con 0x7f71600b1730 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.709+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7160135d30 con 0x7f71600b1730 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.709+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7150001970 con 0x7f71600b1730 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.709+0000 7f715cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.709+0000 7f715cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.709+0000 7f715cff9640 10 monclient(hunting): dump: 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.709 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:31.710 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:31.710 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:31.710 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:31.710 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:31.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.719+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2215217171 0 0) 0x7f7160144d40 con 0x7f716014be90 2024-03-25T19:44:31.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 >> v1:172.21.15.50:6790/0 conn(0x7f71600b1730 legacy=0x7f71601db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 >> v1:172.21.15.50:6789/0 conn(0x7f71600b4d60 legacy=0x7f7160144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 1 monclient: found mon.b 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71601382f0 con 0x7f716014be90 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f715cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 5 monclient: authenticate success, global_id 4375 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 10 monclient: _renew_subs 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7160145670 con 0x7f716014be90 2024-03-25T19:44:31.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 10 monclient: _renew_subs 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7160145920 con 0x7f716014be90 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.720+0000 7f716689d640 1 librados: init done 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.721+0000 7f716689d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.721+0000 7f716689d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.721+0000 7f716689d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:31.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:31.721+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71600627b0 con 0x7f716014be90 2024-03-25T19:44:32.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.006+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71580020a0 con 0x7f716014be90 2024-03-25T19:44:32.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.006+0000 7f715cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.006+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7158003b30 con 0x7f716014be90 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.006+0000 7f715cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.006+0000 7f715cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.006+0000 7f715cff9640 10 monclient: dump: 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:32.007 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:32.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.006+0000 7f715e7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:32.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.007+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f7158003d50 con 0x7f716014be90 2024-03-25T19:44:32.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.008+0000 7f715ffff640 10 monclient: get_auth_request con 0x7f714c06df20 auth_method 0 2024-03-25T19:44:32.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.008+0000 7f715cff9640 10 client.4375.objecter ms_handle_connect 0x7f714c06df20 2024-03-25T19:44:32.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.012+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f7158053750 con 0x7f716014be90 2024-03-25T19:44:32.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.012+0000 7f715cff9640 10 client.4375.objecter ms_dispatch 0x7f7160000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:32.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.012+0000 7f715cff9640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:32.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.012+0000 7f715cff9640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:32.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.012+0000 7f715cff9640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T19:44:32.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.013+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f715804f260 con 0x7f716014be90 2024-03-25T19:44:32.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.013+0000 7f715cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:32.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.013+0000 7f715cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:32.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.248+0000 7f716689d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.248+0000 7f716689d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.248+0000 7f716689d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:32.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.248+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7160069f40 con 0x7f716014be90 2024-03-25T19:44:32.280 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.281+0000 7f715cff9640 1 -- 172.21.15.50:0/1228519950 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f7158051190 con 0x7f716014be90 2024-03-25T19:44:32.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.281+0000 7f715cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.281+0000 7f715cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:32.281 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:32.281 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 10 librados: watch_flush enter 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 10 librados: watch_flush exit 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 10 client.4375.objecter successfully canceled tick 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 >> v1:172.21.15.76:6800/2572556811 conn(0x7f714c06df20 legacy=0x7f714c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 10 monclient: shutdown 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 >> v1:172.21.15.76:6789/0 conn(0x7f716014be90 legacy=0x7f71601e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 shutdown_connections 2024-03-25T19:44:32.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 >> 172.21.15.50:0/1228519950 conn(0x7f71600617a0 msgr2=0x7f71600b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:32.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 shutdown_connections 2024-03-25T19:44:32.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 1 -- 172.21.15.50:0/1228519950 wait complete. 2024-03-25T19:44:32.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:32.284+0000 7f716689d640 1 librados: shutdown 2024-03-25T19:44:32.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:34.540 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:55.046 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.586 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:55.056 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.590 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:55.096 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.594 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:55.064 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.630 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:03:55.136 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.640 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:03:55.110 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.666 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:55.172 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.672 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:55.178 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:34.702 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:03:55.208 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.102 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.218+0000 7f3cf65ef640 10 monclient: get_monmap_and_config 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.218+0000 7f3cf65ef640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.218+0000 7f3cf65ef640 10 monclient: monmap: 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:39.219037+0000 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:39.219037+0000 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:39.218 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:39.219 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:39.219 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.219+0000 7f3cf65ef640 1 Processor -- start 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.219+0000 7f3cf65ef640 1 -- start start 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.219+0000 7f3cf65ef640 10 monclient: init 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf00a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf01430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf0142520 con 0x7f3cf013f710 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf00b28a0 con 0x7f3cf01430c0 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf00b2a30 con 0x7f3cf00a98f0 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.220+0000 7f3cf65ef640 10 monclient(hunting): authenticate will time out at 5054.484225s 2024-03-25T19:44:39.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.240+0000 7f3cf4b8c640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3cf01430c0 0x7f3cf014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:47540/0 (socket says 172.21.15.50:47540) 2024-03-25T19:44:39.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.240+0000 7f3cf4b8c640 1 -- 172.21.15.50:0/460635678 learned_addr learned my addr 172.21.15.50:0/460635678 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.262+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299086390 0 0) 0x7f3cf00b28a0 con 0x7f3cf01430c0 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.262+0000 7f3ceeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.262+0000 7f3ceeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.262+0000 7f3ceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.262+0000 7f3ceeffd640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.262+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cd8003520 con 0x7f3cf01430c0 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.263+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45577537 0 0) 0x7f3cf00b2a30 con 0x7f3cf00a98f0 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.263+0000 7f3ceeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.263+0000 7f3ceeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.263+0000 7f3ceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf00b28a0 con 0x7f3cf00a98f0 2024-03-25T19:44:39.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027507856 0 0) 0x7f3cf0142520 con 0x7f3cf013f710 2024-03-25T19:44:39.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:44:39.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.264+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf00b2a30 con 0x7f3cf013f710 2024-03-25T19:44:39.273 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1079681722 0 0) 0x7f3cd8003520 con 0x7f3cf01430c0 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 >> v1:172.21.15.50:6790/0 conn(0x7f3cf00a98f0 legacy=0x7f3cf00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 >> v1:172.21.15.50:6789/0 conn(0x7f3cf013f710 legacy=0x7f3cf01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 1 monclient: found mon.noname-c 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf00b44c0 con 0x7f3cf01430c0 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3ce8001a10 con 0x7f3cf01430c0 2024-03-25T19:44:39.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3ceeffd640 10 monclient: dump: 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3cf65ef640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:44:39.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.274+0000 7f3cf65ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ce8001b70 con 0x7f3cf01430c0 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3ceeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3ceeffd640 1 -- 172.21.15.50:0/460635678 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3ce8001f80 con 0x7f3cf01430c0 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3ceeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3ceeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3ceeffd640 10 monclient: dump: 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:39.301 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3cf65ef640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:39.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.301+0000 7f3cf65ef640 4 set_mon_vals no callback set 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.302+0000 7f3cf65ef640 10 monclient: shutdown 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.302+0000 7f3cf65ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.302+0000 7f3cf65ef640 1 -- 172.21.15.50:0/460635678 >> v1:172.21.15.76:6789/0 conn(0x7f3cf01430c0 legacy=0x7f3cf014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.302+0000 7f3cf65ef640 1 -- 172.21.15.50:0/460635678 shutdown_connections 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.302+0000 7f3cf65ef640 1 -- 172.21.15.50:0/460635678 >> 172.21.15.50:0/460635678 conn(0x7f3cf00b7ea0 msgr2=0x7f3cf00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.302+0000 7f3cf65ef640 1 -- 172.21.15.50:0/460635678 shutdown_connections 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.302+0000 7f3cf65ef640 1 -- 172.21.15.50:0/460635678 wait complete. 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.303+0000 7f3cf65ef640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.303+0000 7f3cf65ef640 10 monclient: monmap: 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.304+0000 7f3cf65ef640 1 librados: starting msgr at 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.304+0000 7f3cf65ef640 1 librados: starting objecter 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.304+0000 7f3cf65ef640 1 Processor -- start 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.304+0000 7f3cf65ef640 1 -- start start 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.304+0000 7f3cf65ef640 1 librados: setting wanted keys 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.304+0000 7f3cf65ef640 1 librados: calling monclient init 2024-03-25T19:44:39.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.304+0000 7f3cf65ef640 10 monclient: init 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf00a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf01430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf0137190 con 0x7f3cf01430c0 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf01d41e0 con 0x7f3cf013f710 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf01d4430 con 0x7f3cf00a98f0 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.305+0000 7f3cf65ef640 10 monclient(hunting): authenticate will time out at 5054.569664s 2024-03-25T19:44:39.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.315+0000 7f3cef7fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3cf013f710 0x7f3cf01352e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:47570/0 (socket says 172.21.15.50:47570) 2024-03-25T19:44:39.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.315+0000 7f3cef7fe640 1 -- 172.21.15.50:0/2960321114 learned_addr learned my addr 172.21.15.50:0/2960321114 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.320+0000 7f3cecff9640 10 client.?.objecter ms_handle_connect 0x7f3cf013f710 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.320+0000 7f3cecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.320+0000 7f3cecff9640 10 client.?.objecter ms_handle_connect 0x7f3cf01430c0 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.320+0000 7f3cecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.320+0000 7f3cecff9640 10 client.?.objecter ms_handle_connect 0x7f3cf00a98f0 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.320+0000 7f3cecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.328+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152440392 0 0) 0x7f3cf01d4430 con 0x7f3cf00a98f0 2024-03-25T19:44:39.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.328+0000 7f3cecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.328+0000 7f3cecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.328+0000 7f3cecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.328+0000 7f3cecff9640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:44:39.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.328+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc004eb0 con 0x7f3cf00a98f0 2024-03-25T19:44:39.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.332+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195480744 0 0) 0x7f3cf01d41e0 con 0x7f3cf013f710 2024-03-25T19:44:39.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.332+0000 7f3cecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.332+0000 7f3cecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.332+0000 7f3cecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.332+0000 7f3cecff9640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:44:39.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.332+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf01d4430 con 0x7f3cf013f710 2024-03-25T19:44:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.337+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009309186 0 0) 0x7f3cf0137190 con 0x7f3cf01430c0 2024-03-25T19:44:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.337+0000 7f3cecff9640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.337+0000 7f3cecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.337+0000 7f3cecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.337+0000 7f3cecff9640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:44:39.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.337+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf01d41e0 con 0x7f3cf01430c0 2024-03-25T19:44:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.343+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825335687 0 0) 0x7f3cf01d4430 con 0x7f3cf013f710 2024-03-25T19:44:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.343+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cf0137190 con 0x7f3cf013f710 2024-03-25T19:44:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.343+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3ce40019f0 con 0x7f3cf013f710 2024-03-25T19:44:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.343+0000 7f3cecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.343+0000 7f3cecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.343+0000 7f3cecff9640 10 monclient(hunting): dump: 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:39.344 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.344+0000 7f3cecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.352+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906173164 0 0) 0x7f3cf01d41e0 con 0x7f3cf01430c0 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.352+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cf01d4430 con 0x7f3cf01430c0 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.352+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3ce8001e80 con 0x7f3cf01430c0 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.352+0000 7f3cecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.352+0000 7f3cecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.352+0000 7f3cecff9640 10 monclient(hunting): dump: 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:39.352 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:39.353 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:39.353 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:39.353 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:39.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1241496190 0 0) 0x7f3cf0137190 con 0x7f3cf013f710 2024-03-25T19:44:39.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 >> v1:172.21.15.50:6790/0 conn(0x7f3cf00a98f0 legacy=0x7f3cf01d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 >> v1:172.21.15.50:6789/0 conn(0x7f3cf01430c0 legacy=0x7f3cf0130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 10 monclient: _finish_hunting 0 2024-03-25T19:44:39.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 1 monclient: found mon.b 2024-03-25T19:44:39.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:39.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf013bf90 con 0x7f3cf013f710 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 10 monclient: _finish_auth 0 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 10 monclient: _check_auth_tickets 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 10 monclient: _renew_subs 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf01d5060 con 0x7f3cf013f710 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 10 monclient: _renew_subs 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf0139f00 con 0x7f3cf013f710 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.353+0000 7f3cf65ef640 1 librados: init done 2024-03-25T19:44:39.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.354+0000 7f3cf65ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.354+0000 7f3cf65ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.354+0000 7f3cf65ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:39.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.354+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf00627b0 con 0x7f3cf013f710 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.636+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ce4003610 con 0x7f3cf013f710 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.636+0000 7f3cecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.636+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f3ce40039a0 con 0x7f3cf013f710 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.636+0000 7f3cecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.636+0000 7f3cecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.636+0000 7f3cecff9640 10 monclient: dump: 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:39.636 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.636+0000 7f3cee7fc640 4 set_mon_vals no callback set 2024-03-25T19:44:39.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.637+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f3ce4003bc0 con 0x7f3cf013f710 2024-03-25T19:44:39.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.638+0000 7f3ceffff640 10 monclient: get_auth_request con 0x7f3cdc06df80 auth_method 0 2024-03-25T19:44:39.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.638+0000 7f3cecff9640 10 client.4381.objecter ms_handle_connect 0x7f3cdc06df80 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f3ce4054840 con 0x7f3cf013f710 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 10 client.4381.objecter ms_dispatch 0x7f3cf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ce4050350 con 0x7f3cf013f710 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.642+0000 7f3cecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:39.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.904+0000 7f3cf65ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.904+0000 7f3cf65ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.904+0000 7f3cf65ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:44:39.904 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.904+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cf01d45c0 con 0x7f3cf013f710 2024-03-25T19:44:39.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.949+0000 7f3cecff9640 1 -- 172.21.15.50:0/2960321114 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f3ce4052280 con 0x7f3cf013f710 2024-03-25T19:44:39.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.949+0000 7f3cecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.949+0000 7f3cecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:39.949 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:39.949 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:39.951 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 10 librados: watch_flush enter 2024-03-25T19:44:39.951 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 10 librados: watch_flush exit 2024-03-25T19:44:39.951 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T19:44:39.951 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 10 client.4381.objecter successfully canceled tick 2024-03-25T19:44:39.951 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 >> v1:172.21.15.76:6800/2572556811 conn(0x7f3cdc06df80 legacy=0x7f3cdc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 10 monclient: shutdown 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 >> v1:172.21.15.76:6789/0 conn(0x7f3cf013f710 legacy=0x7f3cf01352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 shutdown_connections 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 >> 172.21.15.50:0/2960321114 conn(0x7f3cf00b7ea0 msgr2=0x7f3cf0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 shutdown_connections 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.951+0000 7f3cf65ef640 1 -- 172.21.15.50:0/2960321114 wait complete. 2024-03-25T19:44:39.952 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:39.952+0000 7f3cf65ef640 1 librados: shutdown 2024-03-25T19:44:39.963 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:46.769 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.871+0000 7fb532bb5640 10 monclient: get_monmap_and_config 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.871+0000 7fb532bb5640 10 monclient: build_initial_monmap 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.871+0000 7fb532bb5640 10 monclient: monmap: 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:46.872482+0000 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:46.872482+0000 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:46.871 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:46.872 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.872+0000 7fb532bb5640 1 Processor -- start 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.872+0000 7fb532bb5640 1 -- start start 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.872+0000 7fb532bb5640 10 monclient: init 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 10 monclient(hunting): picked mon.noname-c con 0x7fb52c06adf0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 10 monclient(hunting): picked mon.noname-b con 0x7fb52c0aee10 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 10 monclient(hunting): picked mon.noname-a con 0x7fb52c140860 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:46.873 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c0af690 con 0x7fb52c140860 2024-03-25T19:44:46.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c0b0c70 con 0x7fb52c06adf0 2024-03-25T19:44:46.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c0b0f50 con 0x7fb52c0aee10 2024-03-25T19:44:46.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:46.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb532bb5640 10 monclient(hunting): authenticate will time out at 5062.137109s 2024-03-25T19:44:46.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb530951640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb52c06adf0 0x7fb52c13d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:52160/0 (socket says 172.21.15.50:52160) 2024-03-25T19:44:46.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.873+0000 7fb530951640 1 -- 172.21.15.50:0/695711972 learned_addr learned my addr 172.21.15.50:0/695711972 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:46.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.903+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740765621 0 0) 0x7fb52c0af690 con 0x7fb52c140860 2024-03-25T19:44:46.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.903+0000 7fb52b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.903+0000 7fb52b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.903+0000 7fb52b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.903+0000 7fb52b7fe640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:44:46.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.903+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb51c003520 con 0x7fb52c140860 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1216097885 0 0) 0x7fb51c003520 con 0x7fb52c140860 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 >> v1:172.21.15.50:6790/0 conn(0x7fb52c0aee10 legacy=0x7fb52c140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 >> v1:172.21.15.76:6789/0 conn(0x7fb52c06adf0 legacy=0x7fb52c13d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:44:46.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c13dc40 con 0x7fb52c140860 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb524001a10 con 0x7fb52c140860 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb52b7fe640 10 monclient: dump: 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:46.920 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:46.921 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:46.921 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb532bb5640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:44:46.921 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.919+0000 7fb532bb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb524001be0 con 0x7fb52c140860 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb52b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb52b7fe640 1 -- 172.21.15.50:0/695711972 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb524002030 con 0x7fb52c140860 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb52b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb52b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb52b7fe640 10 monclient: dump: 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:46.926 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb532bb5640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:46.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.926+0000 7fb532bb5640 4 set_mon_vals no callback set 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.927+0000 7fb532bb5640 10 monclient: shutdown 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.927+0000 7fb532bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.927+0000 7fb532bb5640 1 -- 172.21.15.50:0/695711972 >> v1:172.21.15.50:6789/0 conn(0x7fb52c140860 legacy=0x7fb52c14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.927+0000 7fb532bb5640 1 -- 172.21.15.50:0/695711972 shutdown_connections 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.927+0000 7fb532bb5640 1 -- 172.21.15.50:0/695711972 >> 172.21.15.50:0/695711972 conn(0x7fb52c0b4a80 msgr2=0x7fb52c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.927+0000 7fb532bb5640 1 -- 172.21.15.50:0/695711972 shutdown_connections 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.927+0000 7fb532bb5640 1 -- 172.21.15.50:0/695711972 wait complete. 2024-03-25T19:44:46.929 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 10 monclient: build_initial_monmap 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 10 monclient: monmap: 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 1 librados: starting msgr at 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 1 librados: starting objecter 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 1 Processor -- start 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 1 -- start start 2024-03-25T19:44:46.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 1 librados: setting wanted keys 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 1 librados: calling monclient init 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.928+0000 7fb532bb5640 10 monclient: init 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 10 monclient(hunting): picked mon.noname-b con 0x7fb52c06adf0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 10 monclient(hunting): picked mon.noname-a con 0x7fb52c0aee10 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 10 monclient(hunting): picked mon.noname-c con 0x7fb52c137380 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c13c090 con 0x7fb52c0aee10 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c13ca50 con 0x7fb52c06adf0 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c13cd30 con 0x7fb52c137380 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.929+0000 7fb532bb5640 10 monclient(hunting): authenticate will time out at 5062.193640s 2024-03-25T19:44:46.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.940+0000 7fb530951640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb52c06adf0 0x7fb52c13b3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:52182/0 (socket says 172.21.15.50:52182) 2024-03-25T19:44:46.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.940+0000 7fb530951640 1 -- 172.21.15.50:0/2104480998 learned_addr learned my addr 172.21.15.50:0/2104480998 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:46.943 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.944+0000 7fb5297fa640 10 client.?.objecter ms_handle_connect 0x7fb52c06adf0 2024-03-25T19:44:46.943 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.944+0000 7fb5297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.946+0000 7fb5297fa640 10 client.?.objecter ms_handle_connect 0x7fb52c137380 2024-03-25T19:44:46.945 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.946+0000 7fb5297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.949+0000 7fb5297fa640 10 client.?.objecter ms_handle_connect 0x7fb52c0aee10 2024-03-25T19:44:46.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.949+0000 7fb5297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.954 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427425530 0 0) 0x7fb52c13ca50 con 0x7fb52c06adf0 2024-03-25T19:44:46.954 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.954 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.954 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.954 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:44:46.954 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb510004eb0 con 0x7fb52c06adf0 2024-03-25T19:44:46.954 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035488852 0 0) 0x7fb52c13cd30 con 0x7fb52c137380 2024-03-25T19:44:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:44:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.954+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb52c13ca50 con 0x7fb52c137380 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.962+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3075850206 0 0) 0x7fb52c13ca50 con 0x7fb52c137380 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.962+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb52c13cd30 con 0x7fb52c137380 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.962+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb524001ef0 con 0x7fb52c137380 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.962+0000 7fb5297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.962+0000 7fb5297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.962+0000 7fb5297fa640 10 monclient(hunting): dump: 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:46.962 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.963 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:46.963 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:46.963 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:46.963 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:46.963 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:46.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.962+0000 7fb5297fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:46.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.967+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654445420 0 0) 0x7fb52c13c090 con 0x7fb52c0aee10 2024-03-25T19:44:46.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.967+0000 7fb5297fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.967+0000 7fb5297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.967+0000 7fb5297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.967+0000 7fb5297fa640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:44:46.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.967+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb52c13ca50 con 0x7fb52c0aee10 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3433323838 0 0) 0x7fb52c13cd30 con 0x7fb52c137380 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 >> v1:172.21.15.76:6789/0 conn(0x7fb52c06adf0 legacy=0x7fb52c13b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 >> v1:172.21.15.50:6789/0 conn(0x7fb52c0aee10 legacy=0x7fb52c138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 1 monclient: found mon.c 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c137c50 con 0x7fb52c137380 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb5297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:46.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 10 monclient: _renew_subs 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c12df20 con 0x7fb52c137380 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 10 monclient: _renew_subs 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52c1e3d00 con 0x7fb52c137380 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.971+0000 7fb532bb5640 1 librados: init done 2024-03-25T19:44:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.972+0000 7fb532bb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.972+0000 7fb532bb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.972+0000 7fb532bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:46.972+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52c12e160 con 0x7fb52c137380 2024-03-25T19:44:47.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.322+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb524002090 con 0x7fb52c137380 2024-03-25T19:44:47.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.322+0000 7fb5297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.322+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb524003c20 con 0x7fb52c137380 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.322+0000 7fb5297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.322+0000 7fb5297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.322+0000 7fb5297fa640 10 monclient: dump: 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:47.322 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:47.323 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:47.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.322+0000 7fb52affd640 4 set_mon_vals no callback set 2024-03-25T19:44:47.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.323+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fb524019bf0 con 0x7fb52c137380 2024-03-25T19:44:47.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.323+0000 7fb530951640 10 monclient: get_auth_request con 0x7fb51006de00 auth_method 0 2024-03-25T19:44:47.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.323+0000 7fb5297fa640 10 client.4385.objecter ms_handle_connect 0x7fb51006de00 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fb524053720 con 0x7fb52c137380 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 10 client.4385.objecter ms_dispatch 0x7fb52c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb52404f230 con 0x7fb52c137380 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.329+0000 7fb5297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:47.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.587+0000 7fb532bb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.587+0000 7fb532bb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.587+0000 7fb532bb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:44:47.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.587+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb52c06b1d0 con 0x7fb52c137380 2024-03-25T19:44:47.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.612+0000 7fb5297fa640 1 -- 172.21.15.50:0/2104480998 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fb524051160 con 0x7fb52c137380 2024-03-25T19:44:47.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.612+0000 7fb5297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.612+0000 7fb5297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:47.612 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:47.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.614+0000 7fb532bb5640 10 librados: watch_flush enter 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.614+0000 7fb532bb5640 10 librados: watch_flush exit 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.614+0000 7fb532bb5640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.614+0000 7fb532bb5640 10 client.4385.objecter successfully canceled tick 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.614+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 >> v1:172.21.15.76:6800/2572556811 conn(0x7fb51006de00 legacy=0x7fb5100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 10 monclient: shutdown 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 >> v1:172.21.15.50:6790/0 conn(0x7fb52c137380 legacy=0x7fb52c1e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 shutdown_connections 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 >> 172.21.15.50:0/2104480998 conn(0x7fb52c0b4a80 msgr2=0x7fb52c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 shutdown_connections 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 1 -- 172.21.15.50:0/2104480998 wait complete. 2024-03-25T19:44:47.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:47.615+0000 7fb532bb5640 1 librados: shutdown 2024-03-25T19:44:47.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:54.433 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.547+0000 7fc1f54a8640 10 monclient: get_monmap_and_config 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.547+0000 7fc1f54a8640 10 monclient: build_initial_monmap 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.547+0000 7fc1f54a8640 10 monclient: monmap: 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:44:54.549306+0000 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:44:54.549306+0000 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:44:54.548 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:44:54.549 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:54.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.548+0000 7fc1f54a8640 1 Processor -- start 2024-03-25T19:44:54.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.548+0000 7fc1f54a8640 1 -- start start 2024-03-25T19:44:54.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.548+0000 7fc1f54a8640 10 monclient: init 2024-03-25T19:44:54.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:54.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:54.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc1f00a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:54.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc1f013f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:54.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc1f01430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:54.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f0142520 con 0x7fc1f00a98f0 2024-03-25T19:44:54.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f00b28a0 con 0x7fc1f01430c0 2024-03-25T19:44:54.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f00b2a30 con 0x7fc1f013f710 2024-03-25T19:44:54.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:54.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.549+0000 7fc1f54a8640 10 monclient(hunting): authenticate will time out at 5069.813965s 2024-03-25T19:44:54.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.558+0000 7fc1ef7fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc1f01430c0 0x7fc1f014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:36790/0 (socket says 172.21.15.50:36790) 2024-03-25T19:44:54.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.558+0000 7fc1ef7fe640 1 -- 172.21.15.50:0/1605374078 learned_addr learned my addr 172.21.15.50:0/1605374078 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:54.568 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.568+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459219258 0 0) 0x7fc1f00b28a0 con 0x7fc1f01430c0 2024-03-25T19:44:54.568 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.568+0000 7fc1ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.568 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.568+0000 7fc1ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.568+0000 7fc1ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.568+0000 7fc1ee7fc640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:44:54.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.568+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1dc003520 con 0x7fc1f01430c0 2024-03-25T19:44:54.578 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.578+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622587739 0 0) 0x7fc1f0142520 con 0x7fc1f00a98f0 2024-03-25T19:44:54.578 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.578+0000 7fc1ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.578 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.578+0000 7fc1ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.578 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.578+0000 7fc1ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.579 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.578+0000 7fc1ee7fc640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:44:54.579 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.578+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f00b28a0 con 0x7fc1f00a98f0 2024-03-25T19:44:54.582 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248444512 0 0) 0x7fc1dc003520 con 0x7fc1f01430c0 2024-03-25T19:44:54.582 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 >> v1:172.21.15.50:6790/0 conn(0x7fc1f013f710 legacy=0x7fc1f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.582 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 >> v1:172.21.15.50:6789/0 conn(0x7fc1f00a98f0 legacy=0x7fc1f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.582 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:54.582 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 1 monclient: found mon.noname-c 2024-03-25T19:44:54.582 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00b44c0 con 0x7fc1f01430c0 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc1e8001a10 con 0x7fc1f01430c0 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1ee7fc640 10 monclient: dump: 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.583 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:54.584 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:54.584 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:54.584 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:54.584 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:54.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1f54a8640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:44:54.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.581+0000 7fc1f54a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:54.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1e8001b70 con 0x7fc1f01430c0 2024-03-25T19:44:54.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:54.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1ee7fc640 1 -- 172.21.15.50:0/1605374078 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc1e8001f80 con 0x7fc1f01430c0 2024-03-25T19:44:54.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1f54a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1f54a8640 4 set_mon_vals no callback set 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.594+0000 7fc1ee7fc640 10 monclient: dump: 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:54.595 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.595+0000 7fc1f54a8640 10 monclient: shutdown 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.595+0000 7fc1f54a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.595+0000 7fc1f54a8640 1 -- 172.21.15.50:0/1605374078 >> v1:172.21.15.76:6789/0 conn(0x7fc1f01430c0 legacy=0x7fc1f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.595+0000 7fc1f54a8640 1 -- 172.21.15.50:0/1605374078 shutdown_connections 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.595+0000 7fc1f54a8640 1 -- 172.21.15.50:0/1605374078 >> 172.21.15.50:0/1605374078 conn(0x7fc1f00b7ea0 msgr2=0x7fc1f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.595+0000 7fc1f54a8640 1 -- 172.21.15.50:0/1605374078 shutdown_connections 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.595+0000 7fc1f54a8640 1 -- 172.21.15.50:0/1605374078 wait complete. 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 10 monclient: build_initial_monmap 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 10 monclient: monmap: 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:54.598 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 1 librados: starting msgr at 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 1 librados: starting objecter 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 1 Processor -- start 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 1 -- start start 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 1 librados: setting wanted keys 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 1 librados: calling monclient init 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.596+0000 7fc1f54a8640 10 monclient: init 2024-03-25T19:44:54.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc1f00a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc1f013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc1f01430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f0138a70 con 0x7fc1f00a98f0 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f00aba50 con 0x7fc1f013f710 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f01d77a0 con 0x7fc1f01430c0 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:54.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.597+0000 7fc1f54a8640 10 monclient(hunting): authenticate will time out at 5069.861986s 2024-03-25T19:44:54.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.598+0000 7fc1ef7fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fc1f01430c0 0x7fc1f01d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:36322/0 (socket says 172.21.15.50:36322) 2024-03-25T19:44:54.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.598+0000 7fc1ef7fe640 1 -- 172.21.15.50:0/2398018919 learned_addr learned my addr 172.21.15.50:0/2398018919 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:44:54.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.599+0000 7fc1e77fe640 10 client.?.objecter ms_handle_connect 0x7fc1f01430c0 2024-03-25T19:44:54.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.599+0000 7fc1e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.600+0000 7fc1e77fe640 10 client.?.objecter ms_handle_connect 0x7fc1f013f710 2024-03-25T19:44:54.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.600+0000 7fc1e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.666+0000 7fc1e77fe640 10 client.?.objecter ms_handle_connect 0x7fc1f00a98f0 2024-03-25T19:44:54.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.666+0000 7fc1e77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.674+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267478344 0 0) 0x7fc1f0138a70 con 0x7fc1f00a98f0 2024-03-25T19:44:54.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.674+0000 7fc1e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.674+0000 7fc1e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.674+0000 7fc1e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.674+0000 7fc1e77fe640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:44:54.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.674+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1d0004eb0 con 0x7fc1f00a98f0 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.691+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 468265284 0 0) 0x7fc1d0004eb0 con 0x7fc1f00a98f0 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.691+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1f0138a70 con 0x7fc1f00a98f0 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.691+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc1e0001970 con 0x7fc1f00a98f0 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.691+0000 7fc1e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.691+0000 7fc1e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.691+0000 7fc1e77fe640 10 monclient(hunting): dump: 2024-03-25T19:44:54.692 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:54.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.691+0000 7fc1e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:54.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2933135192 0 0) 0x7fc1f0138a70 con 0x7fc1f00a98f0 2024-03-25T19:44:54.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 >> v1:172.21.15.50:6790/0 conn(0x7fc1f01430c0 legacy=0x7fc1f01d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 >> v1:172.21.15.76:6789/0 conn(0x7fc1f013f710 legacy=0x7fc1f01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 1 monclient: found mon.a 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f01d83d0 con 0x7fc1f00a98f0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.706+0000 7fc1e77fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1e77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 5 monclient: authenticate success, global_id 4413 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 monclient: _renew_subs 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f01d41e0 con 0x7fc1f00a98f0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 monclient: _renew_subs 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f01d5110 con 0x7fc1f00a98f0 2024-03-25T19:44:54.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 1 librados: init done 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:54.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:54.707+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f01d7930 con 0x7fc1f00a98f0 2024-03-25T19:44:55.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1e0003610 con 0x7fc1f00a98f0 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc1e0003a60 con 0x7fc1f00a98f0 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1e77fe640 10 monclient: dump: 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:44:55.021 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:44:55.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1edffb640 4 set_mon_vals no callback set 2024-03-25T19:44:55.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.020+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fc1e0003c80 con 0x7fc1f00a98f0 2024-03-25T19:44:55.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.021+0000 7fc1e7fff640 10 monclient: get_auth_request con 0x7fc1d006dc70 auth_method 0 2024-03-25T19:44:55.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.021+0000 7fc1e77fe640 10 client.4413.objecter ms_handle_connect 0x7fc1d006dc70 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fc1e0053670 con 0x7fc1f00a98f0 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 10 client.4413.objecter ms_dispatch 0x7fc1f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1e00b08b0 con 0x7fc1f00a98f0 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:55.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.026+0000 7fc1e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:55.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.295+0000 7fc1f54a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.295+0000 7fc1f54a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.295+0000 7fc1f54a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:44:55.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.295+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1f014f320 con 0x7fc1f00a98f0 2024-03-25T19:44:55.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.319+0000 7fc1e77fe640 1 -- 172.21.15.50:0/2398018919 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fc1e004efb0 con 0x7fc1f00a98f0 2024-03-25T19:44:55.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.319+0000 7fc1e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.319+0000 7fc1e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:55.320 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:44:55.320 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:55.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.321+0000 7fc1f54a8640 10 librados: watch_flush enter 2024-03-25T19:44:55.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.321+0000 7fc1f54a8640 10 librados: watch_flush exit 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.321+0000 7fc1f54a8640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.321+0000 7fc1f54a8640 10 client.4413.objecter successfully canceled tick 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 >> v1:172.21.15.76:6800/2572556811 conn(0x7fc1d006dc70 legacy=0x7fc1d0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 10 monclient: shutdown 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 >> v1:172.21.15.50:6789/0 conn(0x7fc1f00a98f0 legacy=0x7fc1f0138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 shutdown_connections 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 >> 172.21.15.50:0/2398018919 conn(0x7fc1f00b7ea0 msgr2=0x7fc1f013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 shutdown_connections 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 1 -- 172.21.15.50:0/2398018919 wait complete. 2024-03-25T19:44:55.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:44:55.322+0000 7fc1f54a8640 1 librados: shutdown 2024-03-25T19:44:55.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:02.142 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:02.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.246+0000 7fbad669a640 10 monclient: get_monmap_and_config 2024-03-25T19:45:02.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.246+0000 7fbad669a640 10 monclient: build_initial_monmap 2024-03-25T19:45:02.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.246+0000 7fbad669a640 10 monclient: monmap: 2024-03-25T19:45:02.246 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:02.246 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:02.246 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:02.247502+0000 2024-03-25T19:45:02.246 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:02.247502+0000 2024-03-25T19:45:02.247 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:02.247 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:02.247 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:02.247 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:02.247 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:02.247 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:02.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.247+0000 7fbad669a640 1 Processor -- start 2024-03-25T19:45:02.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.247+0000 7fbad669a640 1 -- start start 2024-03-25T19:45:02.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.247+0000 7fbad669a640 10 monclient: init 2024-03-25T19:45:02.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:02.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:02.248 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 10 monclient(hunting): picked mon.noname-a con 0x7fbad00a96e0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:02.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 10 monclient(hunting): picked mon.noname-c con 0x7fbad00b1b10 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:02.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 10 monclient(hunting): picked mon.noname-b con 0x7fbad0146060 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:02.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad0140040 con 0x7fbad00a96e0 2024-03-25T19:45:02.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad00b1fa0 con 0x7fbad00b1b10 2024-03-25T19:45:02.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad00b24c0 con 0x7fbad0146060 2024-03-25T19:45:02.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:02.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.248+0000 7fbad669a640 10 monclient(hunting): authenticate will time out at 5077.511907s 2024-03-25T19:45:02.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.261+0000 7fbacffff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fbad00a96e0 0x7fbad00b0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53200/0 (socket says 172.21.15.50:53200) 2024-03-25T19:45:02.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.261+0000 7fbacffff640 1 -- 172.21.15.50:0/3001006212 learned_addr learned my addr 172.21.15.50:0/3001006212 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:02.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.270+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527961477 0 0) 0x7fbad00b24c0 con 0x7fbad0146060 2024-03-25T19:45:02.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.270+0000 7fbaceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.270+0000 7fbaceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.270+0000 7fbaceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.270+0000 7fbaceffd640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:45:02.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.270+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbab8003520 con 0x7fbad0146060 2024-03-25T19:45:02.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754090954 0 0) 0x7fbad0140040 con 0x7fbad00a96e0 2024-03-25T19:45:02.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:45:02.281 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad00b24c0 con 0x7fbad00a96e0 2024-03-25T19:45:02.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677853781 0 0) 0x7fbab8003520 con 0x7fbad0146060 2024-03-25T19:45:02.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 >> v1:172.21.15.76:6789/0 conn(0x7fbad00b1b10 legacy=0x7fbad0145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.281+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 >> v1:172.21.15.50:6789/0 conn(0x7fbad00a96e0 legacy=0x7fbad00b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 1 monclient: found mon.noname-b 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad00b15b0 con 0x7fbad0146060 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 10 monclient: _finish_auth 0 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbac8001a10 con 0x7fbad0146060 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:02.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbaceffd640 10 monclient: dump: 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbad669a640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:45:02.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.282+0000 7fbad669a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:02.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac8001bf0 con 0x7fbad0146060 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbaceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbaceffd640 1 -- 172.21.15.50:0/3001006212 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbac8002000 con 0x7fbad0146060 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbad669a640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbad669a640 4 set_mon_vals no callback set 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbaceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbaceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.294+0000 7fbaceffd640 10 monclient: dump: 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.294 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:02.295 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:02.295 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:02.295 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:02.295 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:02.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.295+0000 7fbad669a640 10 monclient: shutdown 2024-03-25T19:45:02.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.295+0000 7fbad669a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.295+0000 7fbad669a640 1 -- 172.21.15.50:0/3001006212 >> v1:172.21.15.50:6790/0 conn(0x7fbad0146060 legacy=0x7fbad014dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.295+0000 7fbad669a640 1 -- 172.21.15.50:0/3001006212 shutdown_connections 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.295+0000 7fbad669a640 1 -- 172.21.15.50:0/3001006212 >> 172.21.15.50:0/3001006212 conn(0x7fbad00b7ea0 msgr2=0x7fbad00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.295+0000 7fbad669a640 1 -- 172.21.15.50:0/3001006212 shutdown_connections 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.295+0000 7fbad669a640 1 -- 172.21.15.50:0/3001006212 wait complete. 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.296+0000 7fbad669a640 10 monclient: build_initial_monmap 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.296+0000 7fbad669a640 10 monclient: monmap: 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:02.298 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.296+0000 7fbad669a640 1 librados: starting msgr at 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.296+0000 7fbad669a640 1 librados: starting objecter 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.296+0000 7fbad669a640 1 Processor -- start 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 1 -- start start 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 1 librados: setting wanted keys 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 1 librados: calling monclient init 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient: init 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:02.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient(hunting): picked mon.noname-a con 0x7fbad00a96e0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient(hunting): picked mon.noname-c con 0x7fbad0146060 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient(hunting): picked mon.noname-b con 0x7fbad013dd70 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad013d1d0 con 0x7fbad00a96e0 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad01cd710 con 0x7fbad013dd70 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad01ce090 con 0x7fbad0146060 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:02.300 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.297+0000 7fbad669a640 10 monclient(hunting): authenticate will time out at 5077.561678s 2024-03-25T19:45:02.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.304+0000 7fbacf7fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fbad0146060 0x7fbad013d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:33002/0 (socket says 172.21.15.50:33002) 2024-03-25T19:45:02.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.304+0000 7fbacf7fe640 1 -- 172.21.15.50:0/989432105 learned_addr learned my addr 172.21.15.50:0/989432105 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:02.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.305+0000 7fbaccff9640 10 client.?.objecter ms_handle_connect 0x7fbad0146060 2024-03-25T19:45:02.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.305+0000 7fbaccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.322+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720872830 0 0) 0x7fbad01ce090 con 0x7fbad0146060 2024-03-25T19:45:02.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.322+0000 7fbaccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.322+0000 7fbaccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.322+0000 7fbaccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.322+0000 7fbaccff9640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:45:02.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.322+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbabc004eb0 con 0x7fbad0146060 2024-03-25T19:45:02.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.327+0000 7fbaccff9640 10 client.?.objecter ms_handle_connect 0x7fbad00a96e0 2024-03-25T19:45:02.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.327+0000 7fbaccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.328+0000 7fbaccff9640 10 client.?.objecter ms_handle_connect 0x7fbad013dd70 2024-03-25T19:45:02.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.328+0000 7fbaccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.334 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.335+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87197066 0 0) 0x7fbabc004eb0 con 0x7fbad0146060 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.335+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbad01ce090 con 0x7fbad0146060 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.335+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbac4001a20 con 0x7fbad0146060 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.335+0000 7fbaccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.335+0000 7fbaccff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.335+0000 7fbaccff9640 10 monclient(hunting): dump: 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:02.335 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:02.336 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:02.336 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:02.336 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:02.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.335+0000 7fbaccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112233956 0 0) 0x7fbad013d1d0 con 0x7fbad00a96e0 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:45:02.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbabc004eb0 con 0x7fbad00a96e0 2024-03-25T19:45:02.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942368787 0 0) 0x7fbad01cd710 con 0x7fbad013dd70 2024-03-25T19:45:02.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:45:02.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.343+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad013d1d0 con 0x7fbad013dd70 2024-03-25T19:45:02.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4189722678 0 0) 0x7fbad01ce090 con 0x7fbad0146060 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 >> v1:172.21.15.76:6789/0 conn(0x7fbad013dd70 legacy=0x7fbad01e22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 >> v1:172.21.15.50:6789/0 conn(0x7fbad00a96e0 legacy=0x7fbad01cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 1 monclient: found mon.c 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad0140490 con 0x7fbad0146060 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbaccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 5 monclient: authenticate success, global_id 4397 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 10 monclient: _renew_subs 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:02.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad0140770 con 0x7fbad0146060 2024-03-25T19:45:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 10 monclient: _renew_subs 2024-03-25T19:45:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad013e360 con 0x7fbad0146060 2024-03-25T19:45:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.351+0000 7fbad669a640 1 librados: init done 2024-03-25T19:45:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.352+0000 7fbad669a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.352+0000 7fbad669a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.352+0000 7fbad669a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:02.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.352+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad00a9ac0 con 0x7fbad0146060 2024-03-25T19:45:02.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.688+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbac4001ca0 con 0x7fbad0146060 2024-03-25T19:45:02.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.688+0000 7fbaccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:02.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.688+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbac4003ab0 con 0x7fbad0146060 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.688+0000 7fbaccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.688+0000 7fbaccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.688+0000 7fbaccff9640 10 monclient: dump: 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:02.689 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:02.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.688+0000 7fbace7fc640 4 set_mon_vals no callback set 2024-03-25T19:45:02.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.689+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fbac4003cd0 con 0x7fbad0146060 2024-03-25T19:45:02.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.690+0000 7fbacffff640 10 monclient: get_auth_request con 0x7fbabc06de10 auth_method 0 2024-03-25T19:45:02.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.690+0000 7fbaccff9640 10 client.4397.objecter ms_handle_connect 0x7fbabc06de10 2024-03-25T19:45:02.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fbac4054820 con 0x7fbad0146060 2024-03-25T19:45:02.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 10 client.4397.objecter ms_dispatch 0x7fbad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:02.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:02.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:02.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-25T19:45:02.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbac4050330 con 0x7fbad0146060 2024-03-25T19:45:02.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.696+0000 7fbaccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:02.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.942+0000 7fbad669a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.942+0000 7fbad669a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.943 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.942+0000 7fbad669a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:02.943 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.942+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad00627b0 con 0x7fbad0146060 2024-03-25T19:45:02.977 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.977+0000 7fbaccff9640 1 -- 172.21.15.50:0/989432105 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fbac4052260 con 0x7fbad0146060 2024-03-25T19:45:02.977 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.977+0000 7fbaccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.977 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.977+0000 7fbaccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:02.977 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:02.977 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:02.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 10 librados: watch_flush enter 2024-03-25T19:45:02.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 10 librados: watch_flush exit 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 10 client.4397.objecter successfully canceled tick 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 >> v1:172.21.15.76:6800/2572556811 conn(0x7fbabc06de10 legacy=0x7fbabc0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 10 monclient: shutdown 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 >> v1:172.21.15.50:6790/0 conn(0x7fbad0146060 legacy=0x7fbad013d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 shutdown_connections 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 >> 172.21.15.50:0/989432105 conn(0x7fbad00b7ea0 msgr2=0x7fbad01ce580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.979+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 shutdown_connections 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.980+0000 7fbad669a640 1 -- 172.21.15.50:0/989432105 wait complete. 2024-03-25T19:45:02.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:02.980+0000 7fbad669a640 1 librados: shutdown 2024-03-25T19:45:02.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:09.797 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.910+0000 7f17ad8af640 10 monclient: get_monmap_and_config 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.910+0000 7f17ad8af640 10 monclient: build_initial_monmap 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.910+0000 7f17ad8af640 10 monclient: monmap: 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:09.911313+0000 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:09.911313+0000 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:09.910 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:09.911 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:09.911 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:09.911 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:09.911 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.911+0000 7f17ad8af640 1 Processor -- start 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.911+0000 7f17ad8af640 1 -- start start 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.911+0000 7f17ad8af640 10 monclient: init 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.911+0000 7f17ad8af640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.911+0000 7f17ad8af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 10 monclient(hunting): picked mon.noname-b con 0x7f17a80a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 10 monclient(hunting): picked mon.noname-c con 0x7f17a813f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:09.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 10 monclient(hunting): picked mon.noname-a con 0x7f17a81430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a8142520 con 0x7f17a81430c0 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80b28a0 con 0x7f17a813f710 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80b2a30 con 0x7f17a80a98f0 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17ad8af640 10 monclient(hunting): authenticate will time out at 5085.175760s 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17a6ffd640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f17a80a98f0 0x7f17a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:39838/0 (socket says 172.21.15.50:39838) 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f179ffff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f17a813f710 0x7f17a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:58912/0 (socket says 172.21.15.50:58912) 2024-03-25T19:45:09.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:09.912+0000 7f17a6ffd640 1 -- 172.21.15.50:0/3198344437 learned_addr learned my addr 172.21.15.50:0/3198344437 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.010+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209234089 0 0) 0x7f17a80b2a30 con 0x7f17a80a98f0 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.010+0000 7f17a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.010+0000 7f17a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.010+0000 7f17a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.010+0000 7f17a67fc640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:45:10.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.010+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1794003520 con 0x7f17a80a98f0 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.011+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715807433 0 0) 0x7f17a8142520 con 0x7f17a81430c0 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.011+0000 7f17a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.011+0000 7f17a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.011+0000 7f17a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.011+0000 7f17a67fc640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:45:10.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.011+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a80b2a30 con 0x7f17a81430c0 2024-03-25T19:45:10.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.019+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926399323 0 0) 0x7f17a80b28a0 con 0x7f17a813f710 2024-03-25T19:45:10.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.019+0000 7f17a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.019+0000 7f17a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.019+0000 7f17a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.019+0000 7f17a67fc640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:45:10.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.019+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a8142520 con 0x7f17a813f710 2024-03-25T19:45:10.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1119332284 0 0) 0x7f17a80b2a30 con 0x7f17a81430c0 2024-03-25T19:45:10.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 >> v1:172.21.15.50:6790/0 conn(0x7f17a80a98f0 legacy=0x7f17a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 >> v1:172.21.15.76:6789/0 conn(0x7f17a813f710 legacy=0x7f17a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:10.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 1 monclient: found mon.noname-a 2024-03-25T19:45:10.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:10.020 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a80b44c0 con 0x7f17a81430c0 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f17a00019f0 con 0x7f17a81430c0 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17ad8af640 5 monclient: authenticate success, global_id 4422 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17ad8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17a67fc640 10 monclient: dump: 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.021 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.022 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:10.022 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:10.022 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:10.022 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:10.022 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:10.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.020+0000 7f17ad8af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:10.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17a0001b50 con 0x7f17a81430c0 2024-03-25T19:45:10.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:10.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17a67fc640 1 -- 172.21.15.50:0/3198344437 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f17a0001f60 con 0x7f17a81430c0 2024-03-25T19:45:10.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17a67fc640 10 monclient: dump: 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17ad8af640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:10.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.028+0000 7f17ad8af640 4 set_mon_vals no callback set 2024-03-25T19:45:10.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.029+0000 7f17ad8af640 10 monclient: shutdown 2024-03-25T19:45:10.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.029+0000 7f17ad8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:10.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.029+0000 7f17ad8af640 1 -- 172.21.15.50:0/3198344437 >> v1:172.21.15.50:6789/0 conn(0x7f17a81430c0 legacy=0x7f17a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.029+0000 7f17ad8af640 1 -- 172.21.15.50:0/3198344437 shutdown_connections 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.029+0000 7f17ad8af640 1 -- 172.21.15.50:0/3198344437 >> 172.21.15.50:0/3198344437 conn(0x7f17a80b7ea0 msgr2=0x7f17a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.029+0000 7f17ad8af640 1 -- 172.21.15.50:0/3198344437 shutdown_connections 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.029+0000 7f17ad8af640 1 -- 172.21.15.50:0/3198344437 wait complete. 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 10 monclient: build_initial_monmap 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 10 monclient: monmap: 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:10.030 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:10.031 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 1 librados: starting msgr at 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 1 librados: starting objecter 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 1 Processor -- start 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 1 -- start start 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 1 librados: setting wanted keys 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 1 librados: calling monclient init 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.030+0000 7f17ad8af640 10 monclient: init 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 10 monclient(hunting): picked mon.noname-c con 0x7f17a80a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:10.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 10 monclient(hunting): picked mon.noname-a con 0x7f17a813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:10.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 10 monclient(hunting): picked mon.noname-b con 0x7f17a81430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:10.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a8135bf0 con 0x7f17a813f710 2024-03-25T19:45:10.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a80b6d10 con 0x7f17a81430c0 2024-03-25T19:45:10.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a81317f0 con 0x7f17a80a98f0 2024-03-25T19:45:10.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:10.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.031+0000 7f17ad8af640 10 monclient(hunting): authenticate will time out at 5085.295316s 2024-03-25T19:45:10.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.046+0000 7f17a77fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f17a81430c0 0x7f17a81db370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:58954/0 (socket says 172.21.15.50:58954) 2024-03-25T19:45:10.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.046+0000 7f17a77fe640 1 -- 172.21.15.50:0/3314128168 learned_addr learned my addr 172.21.15.50:0/3314128168 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:10.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.046+0000 7f17a4ff9640 10 client.?.objecter ms_handle_connect 0x7f17a81430c0 2024-03-25T19:45:10.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.046+0000 7f17a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.057+0000 7f17a4ff9640 10 client.?.objecter ms_handle_connect 0x7f17a813f710 2024-03-25T19:45:10.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.057+0000 7f17a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.060+0000 7f17a4ff9640 10 client.?.objecter ms_handle_connect 0x7f17a80a98f0 2024-03-25T19:45:10.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.060+0000 7f17a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:10.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.066+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431228676 0 0) 0x7f17a80b6d10 con 0x7f17a81430c0 2024-03-25T19:45:10.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.066+0000 7f17a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.066+0000 7f17a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.066+0000 7f17a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.066+0000 7f17a4ff9640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:45:10.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.066+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1788004eb0 con 0x7f17a81430c0 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833313174 0 0) 0x7f17a81317f0 con 0x7f17a80a98f0 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a80b6d10 con 0x7f17a80a98f0 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884450473 0 0) 0x7f17a8135bf0 con 0x7f17a813f710 2024-03-25T19:45:10.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:45:10.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:10.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:10.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:45:10.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.085+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a81317f0 con 0x7f17a813f710 2024-03-25T19:45:10.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.100+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3435910133 0 0) 0x7f1788004eb0 con 0x7f17a81430c0 2024-03-25T19:45:10.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.100+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17a8135bf0 con 0x7f17a81430c0 2024-03-25T19:45:10.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.100+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f17a0001fa0 con 0x7f17a81430c0 2024-03-25T19:45:10.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.100+0000 7f17a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.100+0000 7f17a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.100+0000 7f17a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:10.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.101+0000 7f17a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.104+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1107338621 0 0) 0x7f17a80b6d10 con 0x7f17a80a98f0 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.104+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1788004eb0 con 0x7f17a80a98f0 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.104+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1798001950 con 0x7f17a80a98f0 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.104+0000 7f17a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.104+0000 7f17a4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.104+0000 7f17a4ff9640 10 monclient(hunting): dump: 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:10.104 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.105 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.105 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.105 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:10.105 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:10.105 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:10.105 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:10.105 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2269757433 0 0) 0x7f17a8135bf0 con 0x7f17a81430c0 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 >> v1:172.21.15.50:6790/0 conn(0x7f17a80a98f0 legacy=0x7f17a81310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 >> v1:172.21.15.50:6789/0 conn(0x7f17a813f710 legacy=0x7f17a81dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 1 monclient: found mon.b 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:10.116 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a8131980 con 0x7f17a81430c0 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 5 monclient: authenticate success, global_id 4405 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 10 monclient: _renew_subs 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a80b76e0 con 0x7f17a81430c0 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 10 monclient: _renew_subs 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a8135270 con 0x7f17a81430c0 2024-03-25T19:45:10.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.116+0000 7f17ad8af640 1 librados: init done 2024-03-25T19:45:10.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.117+0000 7f17ad8af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.117+0000 7f17ad8af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.117+0000 7f17ad8af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:10.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.117+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a80627b0 con 0x7f17a81430c0 2024-03-25T19:45:10.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.399+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17a0003c20 con 0x7f17a81430c0 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.399+0000 7f17a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.399+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f17a0003f70 con 0x7f17a81430c0 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.399+0000 7f17a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.399+0000 7f17a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.399+0000 7f17a4ff9640 10 monclient: dump: 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:10.399 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:10.400 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:10.400 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:10.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.399+0000 7f17a5ffb640 4 set_mon_vals no callback set 2024-03-25T19:45:10.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.400+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f17a0004190 con 0x7f17a81430c0 2024-03-25T19:45:10.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.400+0000 7f17a6ffd640 10 monclient: get_auth_request con 0x7f178806df70 auth_method 0 2024-03-25T19:45:10.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.401+0000 7f17a4ff9640 10 client.4405.objecter ms_handle_connect 0x7f178806df70 2024-03-25T19:45:10.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.405+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f17a0054ba0 con 0x7f17a81430c0 2024-03-25T19:45:10.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.405+0000 7f17a4ff9640 10 client.4405.objecter ms_dispatch 0x7f17a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:10.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.405+0000 7f17a4ff9640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:10.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.405+0000 7f17a4ff9640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:10.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.405+0000 7f17a4ff9640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-25T19:45:10.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.406+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17a00506b0 con 0x7f17a81430c0 2024-03-25T19:45:10.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.406+0000 7f17a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:10.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.406+0000 7f17a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:10.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.656+0000 7f17ad8af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.656+0000 7f17ad8af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.656+0000 7f17ad8af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:10.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.656+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a8074cc0 con 0x7f17a81430c0 2024-03-25T19:45:10.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.690+0000 7f17a4ff9640 1 -- 172.21.15.50:0/3314128168 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f17a00525e0 con 0x7f17a81430c0 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.690+0000 7f17a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.690+0000 7f17a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:10.690 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 10 librados: watch_flush enter 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 10 librados: watch_flush exit 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 10 client.4405.objecter successfully canceled tick 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 >> v1:172.21.15.76:6800/2572556811 conn(0x7f178806df70 legacy=0x7f1788070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 10 monclient: shutdown 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.692+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 >> v1:172.21.15.76:6789/0 conn(0x7f17a81430c0 legacy=0x7f17a81db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.693+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 shutdown_connections 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.693+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 >> 172.21.15.50:0/3314128168 conn(0x7f17a80b7ea0 msgr2=0x7f17a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:10.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.693+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 shutdown_connections 2024-03-25T19:45:10.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.693+0000 7f17ad8af640 1 -- 172.21.15.50:0/3314128168 wait complete. 2024-03-25T19:45:10.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:10.693+0000 7f17ad8af640 1 librados: shutdown 2024-03-25T19:45:10.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:14.717 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:35.223 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.728 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:35.234 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.776 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:35.282 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.783 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:04:35.289 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.793 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:04:35.299 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.798 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:04:35.269 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.804 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:04:35.275 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.816 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:04:35.322 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:14.832 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:04:35.302 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.512 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:17.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.629+0000 7f63ab64e640 10 monclient: get_monmap_and_config 2024-03-25T19:45:17.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.629+0000 7f63ab64e640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.630+0000 7f63ab64e640 10 monclient: monmap: 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:17.630831+0000 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:17.630831+0000 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:17.630 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.630+0000 7f63ab64e640 1 Processor -- start 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.630+0000 7f63ab64e640 1 -- start start 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.630+0000 7f63ab64e640 10 monclient: init 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 10 monclient(hunting): picked mon.noname-b con 0x7f63a40a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 10 monclient(hunting): picked mon.noname-a con 0x7f63a413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 10 monclient(hunting): picked mon.noname-c con 0x7f63a41430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a4142520 con 0x7f63a413f710 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40b28a0 con 0x7f63a41430c0 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a40b2a30 con 0x7f63a40a98f0 2024-03-25T19:45:17.632 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.633 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.631+0000 7f63ab64e640 10 monclient(hunting): authenticate will time out at 5092.895386s 2024-03-25T19:45:17.646 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.647+0000 7f63a9beb640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f63a41430c0 0x7f63a414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:54938/0 (socket says 172.21.15.50:54938) 2024-03-25T19:45:17.646 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.647+0000 7f63a9beb640 1 -- 172.21.15.50:0/828756068 learned_addr learned my addr 172.21.15.50:0/828756068 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953035362 0 0) 0x7f63a40b28a0 con 0x7f63a41430c0 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6390003520 con 0x7f63a41430c0 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080892251 0 0) 0x7f63a40b2a30 con 0x7f63a40a98f0 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:45:17.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.665+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a40b28a0 con 0x7f63a40a98f0 2024-03-25T19:45:17.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3823742235 0 0) 0x7f6390003520 con 0x7f63a41430c0 2024-03-25T19:45:17.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 >> v1:172.21.15.50:6790/0 conn(0x7f63a40a98f0 legacy=0x7f63a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 >> v1:172.21.15.50:6789/0 conn(0x7f63a413f710 legacy=0x7f63a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 1 monclient: found mon.noname-c 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40b44c0 con 0x7f63a41430c0 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f639c001a10 con 0x7f63a41430c0 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f63ab64e640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f63ab64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.671 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f639bfff640 10 monclient: dump: 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:17.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.670+0000 7f63ab64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:17.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f639c001b70 con 0x7f63a41430c0 2024-03-25T19:45:17.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f639bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f639bfff640 1 -- 172.21.15.50:0/828756068 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f639c001f80 con 0x7f63a41430c0 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f63ab64e640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f63ab64e640 4 set_mon_vals no callback set 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f639bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f639bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.691+0000 7f639bfff640 10 monclient: dump: 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:17.691 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:17.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.692+0000 7f63ab64e640 10 monclient: shutdown 2024-03-25T19:45:17.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.692+0000 7f63ab64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:17.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.692+0000 7f63ab64e640 1 -- 172.21.15.50:0/828756068 >> v1:172.21.15.76:6789/0 conn(0x7f63a41430c0 legacy=0x7f63a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.692+0000 7f63ab64e640 1 -- 172.21.15.50:0/828756068 shutdown_connections 2024-03-25T19:45:17.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.692+0000 7f63ab64e640 1 -- 172.21.15.50:0/828756068 >> 172.21.15.50:0/828756068 conn(0x7f63a40b7ea0 msgr2=0x7f63a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:17.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.692+0000 7f63ab64e640 1 -- 172.21.15.50:0/828756068 shutdown_connections 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.692+0000 7f63ab64e640 1 -- 172.21.15.50:0/828756068 wait complete. 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 10 monclient: monmap: 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:17.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 1 librados: starting msgr at 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 1 librados: starting objecter 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 1 Processor -- start 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 1 -- start start 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 1 librados: setting wanted keys 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 1 librados: calling monclient init 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.693+0000 7f63ab64e640 10 monclient: init 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 10 monclient(hunting): picked mon.noname-c con 0x7f63a40a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 10 monclient(hunting): picked mon.noname-a con 0x7f63a413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 10 monclient(hunting): picked mon.noname-b con 0x7f63a41430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a4137c60 con 0x7f63a413f710 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a4139160 con 0x7f63a41430c0 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a41380f0 con 0x7f63a40a98f0 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.694+0000 7f63ab64e640 10 monclient(hunting): authenticate will time out at 5092.958556s 2024-03-25T19:45:17.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.707+0000 7f63a93ea640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f63a40a98f0 0x7f63a41dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:57418/0 (socket says 172.21.15.50:57418) 2024-03-25T19:45:17.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.707+0000 7f63a93ea640 1 -- 172.21.15.50:0/9024284 learned_addr learned my addr 172.21.15.50:0/9024284 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:17.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.709+0000 7f6399ffb640 10 client.?.objecter ms_handle_connect 0x7f63a40a98f0 2024-03-25T19:45:17.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.709+0000 7f6399ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 10 client.?.objecter ms_handle_connect 0x7f63a41430c0 2024-03-25T19:45:17.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961028968 0 0) 0x7f63a41380f0 con 0x7f63a40a98f0 2024-03-25T19:45:17.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:45:17.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.717+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6388004eb0 con 0x7f63a40a98f0 2024-03-25T19:45:17.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.720+0000 7f6399ffb640 10 client.?.objecter ms_handle_connect 0x7f63a413f710 2024-03-25T19:45:17.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.720+0000 7f6399ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2689947109 0 0) 0x7f6388004eb0 con 0x7f63a40a98f0 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63a41380f0 con 0x7f63a40a98f0 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f63a0001a10 con 0x7f63a40a98f0 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:17.732 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.731+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590481027 0 0) 0x7f63a4139160 con 0x7f63a41430c0 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6388004eb0 con 0x7f63a41430c0 2024-03-25T19:45:17.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513474507 0 0) 0x7f63a4137c60 con 0x7f63a413f710 2024-03-25T19:45:17.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:45:17.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.732+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a4139160 con 0x7f63a413f710 2024-03-25T19:45:17.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 766602856 0 0) 0x7f63a41380f0 con 0x7f63a40a98f0 2024-03-25T19:45:17.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 >> v1:172.21.15.76:6789/0 conn(0x7f63a41430c0 legacy=0x7f63a41e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 >> v1:172.21.15.50:6789/0 conn(0x7f63a413f710 legacy=0x7f63a4133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 1 monclient: found mon.c 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a41397f0 con 0x7f63a40a98f0 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f6399ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 5 monclient: authenticate success, global_id 4409 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 10 monclient: _renew_subs 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a41db280 con 0x7f63a40a98f0 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 10 monclient: _renew_subs 2024-03-25T19:45:17.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:17.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a4133960 con 0x7f63a40a98f0 2024-03-25T19:45:17.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.740+0000 7f63ab64e640 1 librados: init done 2024-03-25T19:45:17.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.741+0000 7f63ab64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.741+0000 7f63ab64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.741+0000 7f63ab64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:17.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:17.741+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a4139a40 con 0x7f63a40a98f0 2024-03-25T19:45:18.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.065+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a0001c90 con 0x7f63a40a98f0 2024-03-25T19:45:18.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.065+0000 7f6399ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:18.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.065+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f63a0003a60 con 0x7f63a40a98f0 2024-03-25T19:45:18.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.065+0000 7f6399ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:18.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.065+0000 7f6399ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:18.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.065+0000 7f6399ffb640 10 monclient: dump: 2024-03-25T19:45:18.065 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:18.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.065+0000 7f639b7fe640 4 set_mon_vals no callback set 2024-03-25T19:45:18.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.066+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f63a0003c80 con 0x7f63a40a98f0 2024-03-25T19:45:18.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.066+0000 7f63a8be9640 10 monclient: get_auth_request con 0x7f638806de10 auth_method 0 2024-03-25T19:45:18.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.066+0000 7f6399ffb640 10 client.4409.objecter ms_handle_connect 0x7f638806de10 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f63a00548b0 con 0x7f63a40a98f0 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 10 client.4409.objecter ms_dispatch 0x7f63a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63a00503c0 con 0x7f63a40a98f0 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:18.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.072+0000 7f6399ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:18.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.316+0000 7f63ab64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.316+0000 7f63ab64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.316+0000 7f63ab64e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:18.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.316+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63a40627b0 con 0x7f63a40a98f0 2024-03-25T19:45:18.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.356+0000 7f6399ffb640 1 -- 172.21.15.50:0/9024284 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f63a00522f0 con 0x7f63a40a98f0 2024-03-25T19:45:18.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.356+0000 7f6399ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:18.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.356+0000 7f6399ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:18.356 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:18.356 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.358+0000 7f63ab64e640 10 librados: watch_flush enter 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.358+0000 7f63ab64e640 10 librados: watch_flush exit 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 10 client.4409.objecter successfully canceled tick 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 >> v1:172.21.15.76:6800/2572556811 conn(0x7f638806de10 legacy=0x7f63880702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 10 monclient: shutdown 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 >> v1:172.21.15.50:6790/0 conn(0x7f63a40a98f0 legacy=0x7f63a41dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 shutdown_connections 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 >> 172.21.15.50:0/9024284 conn(0x7f63a40b7ea0 msgr2=0x7f63a413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 shutdown_connections 2024-03-25T19:45:18.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 1 -- 172.21.15.50:0/9024284 wait complete. 2024-03-25T19:45:18.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:18.359+0000 7f63ab64e640 1 librados: shutdown 2024-03-25T19:45:18.372 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:25.179 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:25.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.296+0000 7f9a437c6640 10 monclient: get_monmap_and_config 2024-03-25T19:45:25.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.296+0000 7f9a437c6640 10 monclient: build_initial_monmap 2024-03-25T19:45:25.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.296+0000 7f9a437c6640 10 monclient: monmap: 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:25.296858+0000 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:25.296858+0000 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:25.296 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.296+0000 7f9a437c6640 1 Processor -- start 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 1 -- start start 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient: init 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a3c0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3c13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3c1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c142520 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0b28a0 con 0x7f9a3c1430c0 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0b2a30 con 0x7f9a3c13f710 2024-03-25T19:45:25.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:25.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.297+0000 7f9a437c6640 10 monclient(hunting): authenticate will time out at 5100.561681s 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.307+0000 7f9a40d61640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f9a3c13f710 0x7f9a3c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:33764/0 (socket says 172.21.15.50:33764) 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.307+0000 7f9a40d61640 1 -- 172.21.15.50:0/2492645525 learned_addr learned my addr 172.21.15.50:0/2492645525 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:25.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.325+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255176640 0 0) 0x7f9a3c0b28a0 con 0x7f9a3c1430c0 2024-03-25T19:45:25.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.325+0000 7f9a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.325+0000 7f9a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.325+0000 7f9a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.325+0000 7f9a33fff640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:45:25.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.325+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a28003520 con 0x7f9a3c1430c0 2024-03-25T19:45:25.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.326+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229402252 0 0) 0x7f9a3c0b2a30 con 0x7f9a3c13f710 2024-03-25T19:45:25.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.326+0000 7f9a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.326+0000 7f9a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.326+0000 7f9a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.326+0000 7f9a33fff640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:45:25.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.326+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a3c0b28a0 con 0x7f9a3c13f710 2024-03-25T19:45:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.329+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342677707 0 0) 0x7f9a3c142520 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.329+0000 7f9a33fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.329+0000 7f9a33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.329+0000 7f9a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.329+0000 7f9a33fff640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:45:25.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.329+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a3c0b2a30 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89507969 0 0) 0x7f9a3c0b2a30 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 >> v1:172.21.15.50:6790/0 conn(0x7f9a3c13f710 legacy=0x7f9a3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 >> v1:172.21.15.76:6789/0 conn(0x7f9a3c1430c0 legacy=0x7f9a3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 1 monclient: found mon.noname-a 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c0b44c0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a437c6640 5 monclient: authenticate success, global_id 4434 2024-03-25T19:45:25.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a437c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9a380014f0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a33fff640 10 monclient: dump: 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:25.342 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:25.343 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:25.343 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:25.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.341+0000 7f9a437c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a380017c0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a33fff640 1 -- 172.21.15.50:0/2492645525 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9a38001b50 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a437c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a437c6640 4 set_mon_vals no callback set 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.366+0000 7f9a33fff640 10 monclient: dump: 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.366 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.367 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:25.367 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:25.367 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:25.367 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:25.367 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:25.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.367+0000 7f9a437c6640 10 monclient: shutdown 2024-03-25T19:45:25.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.367+0000 7f9a437c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:25.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.367+0000 7f9a437c6640 1 -- 172.21.15.50:0/2492645525 >> v1:172.21.15.50:6789/0 conn(0x7f9a3c0a98f0 legacy=0x7f9a3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.367+0000 7f9a437c6640 1 -- 172.21.15.50:0/2492645525 shutdown_connections 2024-03-25T19:45:25.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.367+0000 7f9a437c6640 1 -- 172.21.15.50:0/2492645525 >> 172.21.15.50:0/2492645525 conn(0x7f9a3c0b7ea0 msgr2=0x7f9a3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.367+0000 7f9a437c6640 1 -- 172.21.15.50:0/2492645525 shutdown_connections 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.367+0000 7f9a437c6640 1 -- 172.21.15.50:0/2492645525 wait complete. 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.368+0000 7f9a437c6640 10 monclient: build_initial_monmap 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.368+0000 7f9a437c6640 10 monclient: monmap: 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:25.369 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:25.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.368+0000 7f9a437c6640 1 librados: starting msgr at 2024-03-25T19:45:25.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.368+0000 7f9a437c6640 1 librados: starting objecter 2024-03-25T19:45:25.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.369+0000 7f9a437c6640 1 Processor -- start 2024-03-25T19:45:25.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.369+0000 7f9a437c6640 1 -- start start 2024-03-25T19:45:25.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.369+0000 7f9a437c6640 1 librados: setting wanted keys 2024-03-25T19:45:25.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.369+0000 7f9a437c6640 1 librados: calling monclient init 2024-03-25T19:45:25.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.369+0000 7f9a437c6640 10 monclient: init 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.369+0000 7f9a437c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a3c0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a3c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a3c1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c137fe0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c0ab950 con 0x7f9a3c13f710 2024-03-25T19:45:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3c138860 con 0x7f9a3c1430c0 2024-03-25T19:45:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.370+0000 7f9a437c6640 10 monclient(hunting): authenticate will time out at 5100.633894s 2024-03-25T19:45:25.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.380+0000 7f9a41562640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9a3c0a98f0 0x7f9a3c1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58086/0 (socket says 172.21.15.50:58086) 2024-03-25T19:45:25.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.380+0000 7f9a41562640 1 -- 172.21.15.50:0/1126187908 learned_addr learned my addr 172.21.15.50:0/1126187908 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:25.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.383+0000 7f9a31ffb640 10 client.?.objecter ms_handle_connect 0x7f9a3c13f710 2024-03-25T19:45:25.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.383+0000 7f9a31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.383+0000 7f9a31ffb640 10 client.?.objecter ms_handle_connect 0x7f9a3c0a98f0 2024-03-25T19:45:25.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.383+0000 7f9a31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.390+0000 7f9a31ffb640 10 client.?.objecter ms_handle_connect 0x7f9a3c1430c0 2024-03-25T19:45:25.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.390+0000 7f9a31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.395+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100422326 0 0) 0x7f9a3c0ab950 con 0x7f9a3c13f710 2024-03-25T19:45:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.395+0000 7f9a31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.395+0000 7f9a31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.395+0000 7f9a31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.395+0000 7f9a31ffb640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:45:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.395+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a20004eb0 con 0x7f9a3c13f710 2024-03-25T19:45:25.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.396+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320148613 0 0) 0x7f9a3c137fe0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.396+0000 7f9a31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.396+0000 7f9a31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.396+0000 7f9a31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.396+0000 7f9a31ffb640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:45:25.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.396+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a3c0ab950 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.407+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275185935 0 0) 0x7f9a3c138860 con 0x7f9a3c1430c0 2024-03-25T19:45:25.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.407+0000 7f9a31ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:25.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.407+0000 7f9a31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:25.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.407+0000 7f9a31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:25.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.407+0000 7f9a31ffb640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:45:25.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.407+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a3c137fe0 con 0x7f9a3c1430c0 2024-03-25T19:45:25.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.410+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260392978 0 0) 0x7f9a3c0ab950 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.410+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a3c138860 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.413+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9a380035f0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.413+0000 7f9a31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.413+0000 7f9a31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:25.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.413+0000 7f9a31ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:25.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.413+0000 7f9a31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204018708 0 0) 0x7f9a20004eb0 con 0x7f9a3c13f710 2024-03-25T19:45:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a3c0ab950 con 0x7f9a3c13f710 2024-03-25T19:45:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9a2c001930 con 0x7f9a3c13f710 2024-03-25T19:45:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1900122704 0 0) 0x7f9a3c138860 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 >> v1:172.21.15.50:6790/0 conn(0x7f9a3c1430c0 legacy=0x7f9a3c1d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 >> v1:172.21.15.76:6789/0 conn(0x7f9a3c13f710 legacy=0x7f9a3c1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 monclient: found mon.a 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3c138190 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 10 monclient: _renew_subs 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c138440 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 10 monclient: _renew_subs 2024-03-25T19:45:25.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:25.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c1e30c0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.422+0000 7f9a437c6640 1 librados: init done 2024-03-25T19:45:25.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.424+0000 7f9a437c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.424+0000 7f9a437c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.424+0000 7f9a437c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:25.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.424+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c1386d0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a38003ac0 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9a38003f10 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a31ffb640 10 monclient: dump: 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:25.744 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:25.745 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:25.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a337fe640 4 set_mon_vals no callback set 2024-03-25T19:45:25.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.744+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f9a38004130 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.745+0000 7f9a40d61640 10 monclient: get_auth_request con 0x7f9a2006df80 auth_method 0 2024-03-25T19:45:25.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.745+0000 7f9a31ffb640 10 client.4437.objecter ms_handle_connect 0x7f9a2006df80 2024-03-25T19:45:25.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f9a38054980 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 10 client.4437.objecter ms_dispatch 0x7f9a3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:25.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:25.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:25.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T19:45:25.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a38050490 con 0x7f9a3c0a98f0 2024-03-25T19:45:25.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.750+0000 7f9a31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:25.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.979+0000 7f9a437c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.979+0000 7f9a437c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.979+0000 7f9a437c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:25.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:25.979+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a3c0627b0 con 0x7f9a3c0a98f0 2024-03-25T19:45:26.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.057+0000 7f9a31ffb640 1 -- 172.21.15.50:0/1126187908 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f9a38004460 con 0x7f9a3c0a98f0 2024-03-25T19:45:26.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.057+0000 7f9a31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:26.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.057+0000 7f9a31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:26.057 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:26.057 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.059+0000 7f9a437c6640 10 librados: watch_flush enter 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 10 librados: watch_flush exit 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 10 client.4437.objecter successfully canceled tick 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 >> v1:172.21.15.76:6800/2572556811 conn(0x7f9a2006df80 legacy=0x7f9a20070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 10 monclient: shutdown 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 >> v1:172.21.15.50:6789/0 conn(0x7f9a3c0a98f0 legacy=0x7f9a3c1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:26.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 shutdown_connections 2024-03-25T19:45:26.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 >> 172.21.15.50:0/1126187908 conn(0x7f9a3c0b7ea0 msgr2=0x7f9a3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:26.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 shutdown_connections 2024-03-25T19:45:26.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 1 -- 172.21.15.50:0/1126187908 wait complete. 2024-03-25T19:45:26.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:26.060+0000 7f9a437c6640 1 librados: shutdown 2024-03-25T19:45:26.072 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:32.878 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:32.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.985+0000 7f945bf82640 10 monclient: get_monmap_and_config 2024-03-25T19:45:32.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.985+0000 7f945bf82640 10 monclient: build_initial_monmap 2024-03-25T19:45:32.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.985+0000 7f945bf82640 10 monclient: monmap: 2024-03-25T19:45:32.985 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:32.986551+0000 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:32.986551+0000 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:32.987 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:32.987 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:32.987 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:32.987 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:32.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.986+0000 7f945bf82640 1 Processor -- start 2024-03-25T19:45:32.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.986+0000 7f945bf82640 1 -- start start 2024-03-25T19:45:32.989 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.986+0000 7f945bf82640 10 monclient: init 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 10 monclient(hunting): picked mon.noname-b con 0x7f94540b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 10 monclient(hunting): picked mon.noname-c con 0x7f94540b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 10 monclient(hunting): picked mon.noname-a con 0x7f945414be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94540af940 con 0x7f945414be90 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94540b0a70 con 0x7f94540b1730 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94540b0d50 con 0x7f94540b4d60 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:32.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f945bf82640 10 monclient(hunting): authenticate will time out at 5108.250820s 2024-03-25T19:45:32.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f9459d1e640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f94540b4d60 0x7f945413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:43006/0 (socket says 172.21.15.50:43006) 2024-03-25T19:45:32.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:32.987+0000 7f9459d1e640 1 -- 172.21.15.50:0/1889748169 learned_addr learned my addr 172.21.15.50:0/1889748169 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:33.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.022+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677707880 0 0) 0x7f94540b0a70 con 0x7f94540b1730 2024-03-25T19:45:33.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.022+0000 7f9458d1c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.022+0000 7f9458d1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.022+0000 7f9458d1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.022+0000 7f9458d1c640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:45:33.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.022+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9440003520 con 0x7f94540b1730 2024-03-25T19:45:33.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.052+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095379176 0 0) 0x7f94540af940 con 0x7f945414be90 2024-03-25T19:45:33.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.053+0000 7f9458d1c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.053+0000 7f9458d1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.053+0000 7f9458d1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.053+0000 7f9458d1c640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:45:33.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.053+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94540b0a70 con 0x7f945414be90 2024-03-25T19:45:33.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.069+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292212141 0 0) 0x7f94540b0d50 con 0x7f94540b4d60 2024-03-25T19:45:33.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.069+0000 7f9458d1c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.069+0000 7f9458d1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.069+0000 7f9458d1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.069+0000 7f9458d1c640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:45:33.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.069+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94540af940 con 0x7f94540b4d60 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991605272 0 0) 0x7f94540b0a70 con 0x7f945414be90 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 >> v1:172.21.15.50:6790/0 conn(0x7f94540b4d60 legacy=0x7f945413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 >> v1:172.21.15.76:6789/0 conn(0x7f94540b1730 legacy=0x7f94540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 10 monclient: _finish_hunting 0 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 1 monclient: found mon.noname-a 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94540b1030 con 0x7f945414be90 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 10 monclient: _finish_auth 0 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 10 monclient: _check_auth_tickets 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:33.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f944c0019f0 con 0x7f945414be90 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f945bf82640 5 monclient: authenticate success, global_id 4440 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f945bf82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f9458d1c640 10 monclient: dump: 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:33.072 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:33.073 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:33.073 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:33.073 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:33.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.070+0000 7f945bf82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f944c001b50 con 0x7f945414be90 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f9458d1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f9458d1c640 1 -- 172.21.15.50:0/1889748169 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f944c001f60 con 0x7f945414be90 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f9458d1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f9458d1c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f9458d1c640 10 monclient: dump: 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f945bf82640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:33.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.079+0000 7f945bf82640 4 set_mon_vals no callback set 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.080+0000 7f945bf82640 10 monclient: shutdown 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.080+0000 7f945bf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.080+0000 7f945bf82640 1 -- 172.21.15.50:0/1889748169 >> v1:172.21.15.50:6789/0 conn(0x7f945414be90 legacy=0x7f945414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.080+0000 7f945bf82640 1 -- 172.21.15.50:0/1889748169 shutdown_connections 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.080+0000 7f945bf82640 1 -- 172.21.15.50:0/1889748169 >> 172.21.15.50:0/1889748169 conn(0x7f94540617a0 msgr2=0x7f9454061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.080+0000 7f945bf82640 1 -- 172.21.15.50:0/1889748169 shutdown_connections 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.080+0000 7f945bf82640 1 -- 172.21.15.50:0/1889748169 wait complete. 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 10 monclient: build_initial_monmap 2024-03-25T19:45:33.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 10 monclient: monmap: 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 1 librados: starting msgr at 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 1 librados: starting objecter 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 1 Processor -- start 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 1 -- start start 2024-03-25T19:45:33.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 1 librados: setting wanted keys 2024-03-25T19:45:33.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 1 librados: calling monclient init 2024-03-25T19:45:33.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.081+0000 7f945bf82640 10 monclient: init 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 10 monclient(hunting): picked mon.noname-a con 0x7f94540b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 10 monclient(hunting): picked mon.noname-b con 0x7f94540b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 10 monclient(hunting): picked mon.noname-c con 0x7f945414be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94541327e0 con 0x7f94540b1730 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94541408f0 con 0x7f94540b4d60 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9454140df0 con 0x7f945414be90 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.082+0000 7f945bf82640 10 monclient(hunting): authenticate will time out at 5108.346402s 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.083+0000 7f945951d640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f94540b4d60 0x7f94541319c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:41074/0 (socket says 172.21.15.50:41074) 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.083+0000 7f945951d640 1 -- 172.21.15.50:0/1751233940 learned_addr learned my addr 172.21.15.50:0/1751233940 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.084+0000 7f944a7fc640 10 client.?.objecter ms_handle_connect 0x7f94540b4d60 2024-03-25T19:45:33.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.084+0000 7f944a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.093+0000 7f944a7fc640 10 client.?.objecter ms_handle_connect 0x7f94540b1730 2024-03-25T19:45:33.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.093+0000 7f944a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.102+0000 7f944a7fc640 10 client.?.objecter ms_handle_connect 0x7f945414be90 2024-03-25T19:45:33.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.102+0000 7f944a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:33.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.154+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586815666 0 0) 0x7f94541327e0 con 0x7f94540b1730 2024-03-25T19:45:33.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.154+0000 7f944a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.154+0000 7f944a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.154+0000 7f944a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.154+0000 7f944a7fc640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:45:33.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.154+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9438004eb0 con 0x7f94540b1730 2024-03-25T19:45:33.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.165+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939126715 0 0) 0x7f9454140df0 con 0x7f945414be90 2024-03-25T19:45:33.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.165+0000 7f944a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:33.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.165+0000 7f944a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:33.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.165+0000 7f944a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:33.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.165+0000 7f944a7fc640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:45:33.165 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.165+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94541327e0 con 0x7f945414be90 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.175+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 604839975 0 0) 0x7f94541327e0 con 0x7f945414be90 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.175+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9454140df0 con 0x7f945414be90 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.175+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f944c001e60 con 0x7f945414be90 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.175+0000 7f944a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.175+0000 7f944a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.175+0000 7f944a7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:33.175 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:33.176 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.175+0000 7f944a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:33.179 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.180+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3072576849 0 0) 0x7f9438004eb0 con 0x7f94540b1730 2024-03-25T19:45:33.179 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.180+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94541327e0 con 0x7f94540b1730 2024-03-25T19:45:33.179 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.180+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f94500018f0 con 0x7f94540b1730 2024-03-25T19:45:33.179 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.180+0000 7f944a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.179 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.180+0000 7f944a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.180+0000 7f944a7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:33.180 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3729348921 0 0) 0x7f94541327e0 con 0x7f94540b1730 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 >> v1:172.21.15.50:6790/0 conn(0x7f945414be90 legacy=0x7f94541320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 >> v1:172.21.15.76:6789/0 conn(0x7f94540b4d60 legacy=0x7f94541319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 1 monclient: found mon.a 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:33.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9454140380 con 0x7f94540b1730 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f944a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 10 monclient: _renew_subs 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9454140630 con 0x7f94540b1730 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 10 monclient: _renew_subs 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:33.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94541db620 con 0x7f94540b1730 2024-03-25T19:45:33.194 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.192+0000 7f945bf82640 1 librados: init done 2024-03-25T19:45:33.194 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.193+0000 7f945bf82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.194 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.193+0000 7f945bf82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.194 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.193+0000 7f945bf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:33.194 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.193+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945413cc20 con 0x7f94540b1730 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.532+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94500047e0 con 0x7f94540b1730 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.532+0000 7f944a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.532+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9450004af0 con 0x7f94540b1730 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.532+0000 7f944a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.532+0000 7f944a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.532+0000 7f944a7fc640 10 monclient: dump: 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:33.532 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.532+0000 7f944bfff640 4 set_mon_vals no callback set 2024-03-25T19:45:33.533 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.533+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f945001ab80 con 0x7f94540b1730 2024-03-25T19:45:33.534 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.533+0000 7f945951d640 10 monclient: get_auth_request con 0x7f943806de80 auth_method 0 2024-03-25T19:45:33.534 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.534+0000 7f944a7fc640 10 client.4443.objecter ms_handle_connect 0x7f943806de80 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f94500547f0 con 0x7f94540b1730 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 10 client.4443.objecter ms_dispatch 0x7f9454000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9450050130 con 0x7f94540b1730 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:33.541 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.541+0000 7f944a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:33.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.800+0000 7f945bf82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.801+0000 7f945bf82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.801+0000 7f945bf82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:33.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.801+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9454140a80 con 0x7f94540b1730 2024-03-25T19:45:33.848 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.848+0000 7f944a7fc640 1 -- 172.21.15.50:0/1751233940 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f9450052230 con 0x7f94540b1730 2024-03-25T19:45:33.848 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.848+0000 7f944a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.848 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.848+0000 7f944a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:33.848 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:33.848 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.850+0000 7f945bf82640 10 librados: watch_flush enter 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 10 librados: watch_flush exit 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 10 client.4443.objecter successfully canceled tick 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 >> v1:172.21.15.76:6800/2572556811 conn(0x7f943806de80 legacy=0x7f9438070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 10 monclient: shutdown 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 >> v1:172.21.15.50:6789/0 conn(0x7f94540b1730 legacy=0x7f94541dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 shutdown_connections 2024-03-25T19:45:33.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 >> 172.21.15.50:0/1751233940 conn(0x7f94540617a0 msgr2=0x7f9454061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:33.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 shutdown_connections 2024-03-25T19:45:33.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 1 -- 172.21.15.50:0/1751233940 wait complete. 2024-03-25T19:45:33.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:33.851+0000 7f945bf82640 1 librados: shutdown 2024-03-25T19:45:33.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:34.789 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:55.295 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.794 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:55.300 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.836 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:04:55.342 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.846 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:04:55.317 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.860 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:04:55.331 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.875 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:04:55.381 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.886 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:04:55.357 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.888 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:04:55.394 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:34.919 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:04:55.424 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:40.670 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:40.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.786+0000 7fa4e1d0e640 10 monclient: get_monmap_and_config 2024-03-25T19:45:40.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.786+0000 7fa4e1d0e640 10 monclient: build_initial_monmap 2024-03-25T19:45:40.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.786+0000 7fa4e1d0e640 10 monclient: monmap: 2024-03-25T19:45:40.787 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:40.787 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:40.787 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:40.788390+0000 2024-03-25T19:45:40.787 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:40.788390+0000 2024-03-25T19:45:40.788 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:40.788 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:40.788 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:40.788 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:40.788 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:40.788 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:40.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.787+0000 7fa4e1d0e640 1 Processor -- start 2024-03-25T19:45:40.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.787+0000 7fa4e1d0e640 1 -- start start 2024-03-25T19:45:40.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.787+0000 7fa4e1d0e640 10 monclient: init 2024-03-25T19:45:40.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.787+0000 7fa4e1d0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:40.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.787+0000 7fa4e1d0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:40.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.787+0000 7fa4e1d0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4dc0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:40.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.788+0000 7fa4e1d0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4dc13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:40.790 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.788+0000 7fa4e1d0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4dc1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:40.790 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.788+0000 7fa4e1d0e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc142520 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.790 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.788+0000 7fa4e1d0e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0b28a0 con 0x7fa4dc1430c0 2024-03-25T19:45:40.790 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.788+0000 7fa4e1d0e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0b2a30 con 0x7fa4dc13f710 2024-03-25T19:45:40.790 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.788+0000 7fa4e1d0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:40.790 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.788+0000 7fa4e1d0e640 10 monclient(hunting): authenticate will time out at 5116.052767s 2024-03-25T19:45:40.797 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.796+0000 7fa4dbfff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa4dc1430c0 0x7fa4dc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:41724/0 (socket says 172.21.15.50:41724) 2024-03-25T19:45:40.797 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.796+0000 7fa4daffd640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fa4dc13f710 0x7fa4dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:58548/0 (socket says 172.21.15.50:58548) 2024-03-25T19:45:40.797 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.796+0000 7fa4dbfff640 1 -- 172.21.15.50:0/1414649934 learned_addr learned my addr 172.21.15.50:0/1414649934 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.811+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266698164 0 0) 0x7fa4dc0b2a30 con 0x7fa4dc13f710 2024-03-25T19:45:40.811 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.811+0000 7fa4da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.811+0000 7fa4da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.811+0000 7fa4da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.811+0000 7fa4da7fc640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:45:40.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.811+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c4003520 con 0x7fa4dc13f710 2024-03-25T19:45:40.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.814+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005157335 0 0) 0x7fa4dc142520 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.814+0000 7fa4da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.814+0000 7fa4da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.814+0000 7fa4da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.814+0000 7fa4da7fc640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:45:40.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.814+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc0b2a30 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.815+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114033467 0 0) 0x7fa4dc0b28a0 con 0x7fa4dc1430c0 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.815+0000 7fa4da7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.815+0000 7fa4da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.815+0000 7fa4da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.815+0000 7fa4da7fc640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.815+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc142520 con 0x7fa4dc1430c0 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4062083246 0 0) 0x7fa4dc142520 con 0x7fa4dc1430c0 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 >> v1:172.21.15.50:6790/0 conn(0x7fa4dc13f710 legacy=0x7fa4dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 >> v1:172.21.15.50:6789/0 conn(0x7fa4dc0a98f0 legacy=0x7fa4dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 1 monclient: found mon.noname-c 2024-03-25T19:45:40.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc0b44c0 con 0x7fa4dc1430c0 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fa4d4001930 con 0x7fa4dc1430c0 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4da7fc640 10 monclient: dump: 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.835 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4e1d0e640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:45:40.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.833+0000 7fa4e1d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:40.849 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4d4001a90 con 0x7fa4dc1430c0 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4da7fc640 1 -- 172.21.15.50:0/1414649934 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fa4d4001ea0 con 0x7fa4dc1430c0 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4e1d0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4e1d0e640 4 set_mon_vals no callback set 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.849+0000 7fa4da7fc640 10 monclient: dump: 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.850 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:40.851 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:40.851 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:40.851 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:40.851 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 10 monclient: shutdown 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/1414649934 >> v1:172.21.15.76:6789/0 conn(0x7fa4dc1430c0 legacy=0x7fa4dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/1414649934 shutdown_connections 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/1414649934 >> 172.21.15.50:0/1414649934 conn(0x7fa4dc0b7ea0 msgr2=0x7fa4dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/1414649934 shutdown_connections 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/1414649934 wait complete. 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 10 monclient: build_initial_monmap 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.850+0000 7fa4e1d0e640 10 monclient: monmap: 2024-03-25T19:45:40.853 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.851+0000 7fa4e1d0e640 1 librados: starting msgr at 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.851+0000 7fa4e1d0e640 1 librados: starting objecter 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.851+0000 7fa4e1d0e640 1 Processor -- start 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.851+0000 7fa4e1d0e640 1 -- start start 2024-03-25T19:45:40.854 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.851+0000 7fa4e1d0e640 1 librados: setting wanted keys 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.851+0000 7fa4e1d0e640 1 librados: calling monclient init 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.851+0000 7fa4e1d0e640 10 monclient: init 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa4dc0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa4dc13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa4dc1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc1d7800 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc1d83a0 con 0x7fa4dc1430c0 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0b04d0 con 0x7fa4dc13f710 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:40.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.852+0000 7fa4e1d0e640 10 monclient(hunting): authenticate will time out at 5116.116842s 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.867+0000 7fa4daffd640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fa4dc13f710 0x7fa4dc138c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:58600/0 (socket says 172.21.15.50:58600) 2024-03-25T19:45:40.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.867+0000 7fa4daffd640 1 -- 172.21.15.50:0/3472908984 learned_addr learned my addr 172.21.15.50:0/3472908984 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:40.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.869+0000 7fa4c3fff640 10 client.?.objecter ms_handle_connect 0x7fa4dc1430c0 2024-03-25T19:45:40.870 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.869+0000 7fa4c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.870+0000 7fa4c3fff640 10 client.?.objecter ms_handle_connect 0x7fa4dc13f710 2024-03-25T19:45:40.871 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.870+0000 7fa4c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.873+0000 7fa4c3fff640 10 client.?.objecter ms_handle_connect 0x7fa4dc0a98f0 2024-03-25T19:45:40.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.873+0000 7fa4c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.880+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331364833 0 0) 0x7fa4dc1d83a0 con 0x7fa4dc1430c0 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.880+0000 7fa4c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.880+0000 7fa4c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.880+0000 7fa4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.880+0000 7fa4c3fff640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:45:40.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.880+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c8004eb0 con 0x7fa4dc1430c0 2024-03-25T19:45:40.885 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.885+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450716290 0 0) 0x7fa4dc0b04d0 con 0x7fa4dc13f710 2024-03-25T19:45:40.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.885+0000 7fa4c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.885+0000 7fa4c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.885+0000 7fa4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.885+0000 7fa4c3fff640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:45:40.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.885+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc1d83a0 con 0x7fa4dc13f710 2024-03-25T19:45:40.889 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.888+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740875632 0 0) 0x7fa4dc1d7800 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.889 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.888+0000 7fa4c3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.889 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.888+0000 7fa4c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.889 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.888+0000 7fa4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.889 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.888+0000 7fa4c3fff640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:45:40.889 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.888+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc0b04d0 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.901+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1710875055 0 0) 0x7fa4dc0b04d0 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.901+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4dc1d7800 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.901+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fa4cc001930 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.901+0000 7fa4c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.902 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.901+0000 7fa4c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.901+0000 7fa4c3fff640 10 monclient(hunting): dump: 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:40.903 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.901+0000 7fa4c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:40.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.908+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 88313894 0 0) 0x7fa4dc1d83a0 con 0x7fa4dc13f710 2024-03-25T19:45:40.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.908+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4dc0b04d0 con 0x7fa4dc13f710 2024-03-25T19:45:40.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.908+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fa4d0001950 con 0x7fa4dc13f710 2024-03-25T19:45:40.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.908+0000 7fa4c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.908+0000 7fa4c3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:40.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.908+0000 7fa4c3fff640 10 monclient(hunting): dump: 2024-03-25T19:45:40.909 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:40.910 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:40.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4016722372 0 0) 0x7fa4dc1d7800 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 >> v1:172.21.15.50:6790/0 conn(0x7fa4dc13f710 legacy=0x7fa4dc138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.913 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 >> v1:172.21.15.76:6789/0 conn(0x7fa4dc1430c0 legacy=0x7fa4dc139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 1 monclient: found mon.a 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc139a30 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 5 monclient: authenticate success, global_id 4449 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 10 monclient: _renew_subs 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:40.914 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc0b0740 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:40.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 10 monclient: _renew_subs 2024-03-25T19:45:40.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:40.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc1d7540 con 0x7fa4dc0a98f0 2024-03-25T19:45:40.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.912+0000 7fa4e1d0e640 1 librados: init done 2024-03-25T19:45:40.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.914+0000 7fa4e1d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.916 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.914+0000 7fa4e1d0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.916 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.914+0000 7fa4e1d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:40.916 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:40.914+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc14e1e0 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4cc003610 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fa4cc001d80 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4c3fff640 10 monclient: dump: 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:41.237 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4d9ffb640 4 set_mon_vals no callback set 2024-03-25T19:45:41.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.236+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fa4cc019790 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.237+0000 7fa4daffd640 10 monclient: get_auth_request con 0x7fa4c806df80 auth_method 0 2024-03-25T19:45:41.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.237+0000 7fa4c3fff640 10 client.4449.objecter ms_handle_connect 0x7fa4c806df80 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fa4cc053510 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 10 client.4449.objecter ms_dispatch 0x7fa4dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4cc04ee50 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:41.244 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.243+0000 7fa4c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:41.503 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.502+0000 7fa4e1d0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.503 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.502+0000 7fa4e1d0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.503 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.502+0000 7fa4e1d0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:45:41.503 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.502+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4dc0627b0 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.535 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.534+0000 7fa4c3fff640 1 -- 172.21.15.50:0/3472908984 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fa4cc050f50 con 0x7fa4dc0a98f0 2024-03-25T19:45:41.535 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.534+0000 7fa4c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:41.535 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.534+0000 7fa4c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:41.535 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:41.535 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.536+0000 7fa4e1d0e640 10 librados: watch_flush enter 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.536+0000 7fa4e1d0e640 10 librados: watch_flush exit 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.536+0000 7fa4e1d0e640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.536+0000 7fa4e1d0e640 10 client.4449.objecter successfully canceled tick 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 >> v1:172.21.15.76:6800/2572556811 conn(0x7fa4c806df80 legacy=0x7fa4c8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 10 monclient: shutdown 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 >> v1:172.21.15.50:6789/0 conn(0x7fa4dc0a98f0 legacy=0x7fa4dc1d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 shutdown_connections 2024-03-25T19:45:41.538 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 >> 172.21.15.50:0/3472908984 conn(0x7fa4dc0b7ea0 msgr2=0x7fa4dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:41.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 shutdown_connections 2024-03-25T19:45:41.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 1 -- 172.21.15.50:0/3472908984 wait complete. 2024-03-25T19:45:41.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:41.537+0000 7fa4e1d0e640 1 librados: shutdown 2024-03-25T19:45:41.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:48.356 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:48.473 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.474+0000 7f46ab54b640 10 monclient: get_monmap_and_config 2024-03-25T19:45:48.473 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.474+0000 7f46ab54b640 10 monclient: build_initial_monmap 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.474+0000 7f46ab54b640 10 monclient: monmap: 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:48.474882+0000 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:48.474882+0000 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:48.474 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.474+0000 7f46ab54b640 1 Processor -- start 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.474+0000 7f46ab54b640 1 -- start start 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.474+0000 7f46ab54b640 10 monclient: init 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 10 monclient(hunting): picked mon.noname-c con 0x7f46a40a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 10 monclient(hunting): picked mon.noname-b con 0x7f46a413f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 10 monclient(hunting): picked mon.noname-a con 0x7f46a41430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a4142520 con 0x7f46a41430c0 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40b28a0 con 0x7f46a40a98f0 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40b2a30 con 0x7f46a413f710 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:48.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.475+0000 7f46ab54b640 10 monclient(hunting): authenticate will time out at 5123.739374s 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.483+0000 7f46a8ae6640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f46a413f710 0x7f46a41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:34162/0 (socket says 172.21.15.50:34162) 2024-03-25T19:45:48.483 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.484+0000 7f46a8ae6640 1 -- 172.21.15.50:0/2972860478 learned_addr learned my addr 172.21.15.50:0/2972860478 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:48.498 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.499+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189126517 0 0) 0x7f46a40b2a30 con 0x7f46a413f710 2024-03-25T19:45:48.498 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.499+0000 7f469bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.499+0000 7f469bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.499+0000 7f469bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.499+0000 7f469bfff640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:45:48.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.499+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4690003520 con 0x7f46a413f710 2024-03-25T19:45:48.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.509+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812215784 0 0) 0x7f46a40b28a0 con 0x7f46a40a98f0 2024-03-25T19:45:48.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.509+0000 7f469bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.509+0000 7f469bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.509+0000 7f469bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.509+0000 7f469bfff640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:45:48.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.509+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a40b2a30 con 0x7f46a40a98f0 2024-03-25T19:45:48.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.512+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549234748 0 0) 0x7f46a4142520 con 0x7f46a41430c0 2024-03-25T19:45:48.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.512+0000 7f469bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.512+0000 7f469bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.512+0000 7f469bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.512+0000 7f469bfff640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:45:48.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.512+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a40b28a0 con 0x7f46a41430c0 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913181244 0 0) 0x7f4690003520 con 0x7f46a413f710 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 >> v1:172.21.15.76:6789/0 conn(0x7f46a40a98f0 legacy=0x7f46a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 >> v1:172.21.15.50:6789/0 conn(0x7f46a41430c0 legacy=0x7f46a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 1 monclient: found mon.noname-b 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40b44c0 con 0x7f46a413f710 2024-03-25T19:45:48.517 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 10 monclient: _finish_auth 0 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4694001690 con 0x7f46a413f710 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f469bfff640 10 monclient: dump: 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:48.518 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:48.519 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:48.519 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:48.519 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f46ab54b640 5 monclient: authenticate success, global_id 4430 2024-03-25T19:45:48.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.517+0000 7f46ab54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:48.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46940017f0 con 0x7f46a413f710 2024-03-25T19:45:48.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f469bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:48.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f469bfff640 1 -- 172.21.15.50:0/2972860478 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4694001c00 con 0x7f46a413f710 2024-03-25T19:45:48.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f46ab54b640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f46ab54b640 4 set_mon_vals no callback set 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f469bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f469bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f469bfff640 10 monclient: dump: 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:48.540 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:48.541 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f46ab54b640 10 monclient: shutdown 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f46ab54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f46ab54b640 1 -- 172.21.15.50:0/2972860478 >> v1:172.21.15.50:6790/0 conn(0x7f46a413f710 legacy=0x7f46a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f46ab54b640 1 -- 172.21.15.50:0/2972860478 shutdown_connections 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.540+0000 7f46ab54b640 1 -- 172.21.15.50:0/2972860478 >> 172.21.15.50:0/2972860478 conn(0x7f46a40b7ea0 msgr2=0x7f46a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.541+0000 7f46ab54b640 1 -- 172.21.15.50:0/2972860478 shutdown_connections 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.541+0000 7f46ab54b640 1 -- 172.21.15.50:0/2972860478 wait complete. 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.541+0000 7f46ab54b640 10 monclient: build_initial_monmap 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.541+0000 7f46ab54b640 10 monclient: monmap: 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:48.543 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 librados: starting msgr at 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 librados: starting objecter 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 Processor -- start 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 -- start start 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 librados: setting wanted keys 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 librados: calling monclient init 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient: init 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:48.544 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient(hunting): picked mon.noname-a con 0x7f46a40a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:48.545 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient(hunting): picked mon.noname-b con 0x7f46a413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:48.545 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient(hunting): picked mon.noname-c con 0x7f46a41430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:48.545 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40aba00 con 0x7f46a40a98f0 2024-03-25T19:45:48.545 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41d7490 con 0x7f46a413f710 2024-03-25T19:45:48.545 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a41d8360 con 0x7f46a41430c0 2024-03-25T19:45:48.545 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:48.545 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.542+0000 7f46ab54b640 10 monclient(hunting): authenticate will time out at 5123.806625s 2024-03-25T19:45:48.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.548+0000 7f46a92e7640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f46a40a98f0 0x7f46a41386e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34138/0 (socket says 172.21.15.50:34138) 2024-03-25T19:45:48.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.548+0000 7f46a92e7640 1 -- 172.21.15.50:0/1640714602 learned_addr learned my addr 172.21.15.50:0/1640714602 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:48.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.552+0000 7f4699ffb640 10 client.?.objecter ms_handle_connect 0x7f46a40a98f0 2024-03-25T19:45:48.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.552+0000 7f4699ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.558+0000 7f4699ffb640 10 client.?.objecter ms_handle_connect 0x7f46a413f710 2024-03-25T19:45:48.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.558+0000 7f4699ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.562 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.563+0000 7f4699ffb640 10 client.?.objecter ms_handle_connect 0x7f46a41430c0 2024-03-25T19:45:48.563 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.563+0000 7f4699ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:48.563 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.564+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031461249 0 0) 0x7f46a41d7490 con 0x7f46a413f710 2024-03-25T19:45:48.563 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.564+0000 7f4699ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.564+0000 7f4699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.564+0000 7f4699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.564+0000 7f4699ffb640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:45:48.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.564+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4688004eb0 con 0x7f46a413f710 2024-03-25T19:45:48.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.566+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070556093 0 0) 0x7f46a40aba00 con 0x7f46a40a98f0 2024-03-25T19:45:48.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.566+0000 7f4699ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.566+0000 7f4699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.566+0000 7f4699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.566+0000 7f4699ffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:45:48.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.566+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a41d7490 con 0x7f46a40a98f0 2024-03-25T19:45:48.580 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.580+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852131439 0 0) 0x7f46a41d8360 con 0x7f46a41430c0 2024-03-25T19:45:48.580 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.580+0000 7f4699ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:48.580 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.580+0000 7f4699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:48.580 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.580+0000 7f4699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:48.580 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.580+0000 7f4699ffb640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:45:48.580 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.580+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a40aba00 con 0x7f46a41430c0 2024-03-25T19:45:48.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.595+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 92024183 0 0) 0x7f4688004eb0 con 0x7f46a413f710 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.595+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a41d8360 con 0x7f46a413f710 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.595+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4694001b00 con 0x7f46a413f710 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.595+0000 7f4699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.595+0000 7f4699ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.595+0000 7f4699ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:48.595 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:48.596 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:48.596 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:48.596 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.595+0000 7f4699ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:48.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.596+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707978010 0 0) 0x7f46a40aba00 con 0x7f46a41430c0 2024-03-25T19:45:48.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.596+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4688004eb0 con 0x7f46a41430c0 2024-03-25T19:45:48.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.596+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f469c001bd0 con 0x7f46a41430c0 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.596+0000 7f4699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.596+0000 7f4699ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.596+0000 7f4699ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:48.597 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.601+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1337451974 0 0) 0x7f46a41d7490 con 0x7f46a40a98f0 2024-03-25T19:45:48.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.601+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a40aba00 con 0x7f46a40a98f0 2024-03-25T19:45:48.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.601+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f46a00019b0 con 0x7f46a40a98f0 2024-03-25T19:45:48.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.601+0000 7f4699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.601+0000 7f4699ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.601+0000 7f4699ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:48.601 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1343954874 0 0) 0x7f46a41d8360 con 0x7f46a413f710 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 >> v1:172.21.15.50:6790/0 conn(0x7f46a41430c0 legacy=0x7f46a4133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 >> v1:172.21.15.50:6789/0 conn(0x7f46a40a98f0 legacy=0x7f46a41386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 1 monclient: found mon.b 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40abc30 con 0x7f46a413f710 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f4699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:48.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 10 monclient: _renew_subs 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a4137940 con 0x7f46a413f710 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 10 monclient: _renew_subs 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a4135220 con 0x7f46a413f710 2024-03-25T19:45:48.609 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.608+0000 7f46ab54b640 1 librados: init done 2024-03-25T19:45:48.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.609+0000 7f46ab54b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.609+0000 7f46ab54b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.609+0000 7f46ab54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:48.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.609+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a40627b0 con 0x7f46a413f710 2024-03-25T19:45:48.921 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.921+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4694001220 con 0x7f46a413f710 2024-03-25T19:45:48.921 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.921+0000 7f4699ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:48.921 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.921+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4694004c30 con 0x7f46a413f710 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.921+0000 7f4699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.921+0000 7f4699ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.921+0000 7f4699ffb640 10 monclient: dump: 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:48.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.921+0000 7f469b7fe640 4 set_mon_vals no callback set 2024-03-25T19:45:48.923 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.922+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f469401ac00 con 0x7f46a413f710 2024-03-25T19:45:48.923 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.923+0000 7f46a92e7640 10 monclient: get_auth_request con 0x7f468806e0f0 auth_method 0 2024-03-25T19:45:48.923 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.923+0000 7f4699ffb640 10 client.4435.objecter ms_handle_connect 0x7f468806e0f0 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.927+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f46940548d0 con 0x7f46a413f710 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.927+0000 7f4699ffb640 10 client.4435.objecter ms_dispatch 0x7f46a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.927+0000 7f4699ffb640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.927+0000 7f4699ffb640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.927+0000 7f4699ffb640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.927+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4694050210 con 0x7f46a413f710 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.928+0000 7f4699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.927 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:48.928+0000 7f4699ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:49.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.157+0000 7f46ab54b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.157+0000 7f46ab54b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.157+0000 7f46ab54b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:45:49.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.157+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46a4069f40 con 0x7f46a413f710 2024-03-25T19:45:49.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.275+0000 7f4699ffb640 1 -- 172.21.15.50:0/1640714602 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f4694052310 con 0x7f46a413f710 2024-03-25T19:45:49.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.275+0000 7f4699ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:49.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.275+0000 7f4699ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:49.275 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:49.275 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.277+0000 7f46ab54b640 10 librados: watch_flush enter 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.277+0000 7f46ab54b640 10 librados: watch_flush exit 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.277+0000 7f46ab54b640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.277+0000 7f46ab54b640 10 client.4435.objecter successfully canceled tick 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 >> v1:172.21.15.76:6800/2572556811 conn(0x7f468806e0f0 legacy=0x7f46880705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 10 monclient: shutdown 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 >> v1:172.21.15.76:6789/0 conn(0x7f46a413f710 legacy=0x7f46a41d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 shutdown_connections 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 >> 172.21.15.50:0/1640714602 conn(0x7f46a40b7ea0 msgr2=0x7f46a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:49.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 shutdown_connections 2024-03-25T19:45:49.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 1 -- 172.21.15.50:0/1640714602 wait complete. 2024-03-25T19:45:49.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:49.278+0000 7f46ab54b640 1 librados: shutdown 2024-03-25T19:45:49.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:54.877 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:05:15.383 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.878 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:05:15.348 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.881 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:05:15.352 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.886 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:05:15.392 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.893 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:05:15.363 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.920 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:05:15.426 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.939 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:05:15.445 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.947 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:05:15.452 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:54.982 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:05:15.487 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:56.096 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:56.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.214+0000 7f558fd54640 10 monclient: get_monmap_and_config 2024-03-25T19:45:56.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.214+0000 7f558fd54640 10 monclient: build_initial_monmap 2024-03-25T19:45:56.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.214+0000 7f558fd54640 10 monclient: monmap: 2024-03-25T19:45:56.214 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:56.214 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:45:56.215668+0000 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:45:56.215668+0000 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:45:56.215 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.215+0000 7f558fd54640 1 Processor -- start 2024-03-25T19:45:56.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.215+0000 7f558fd54640 1 -- start start 2024-03-25T19:45:56.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.215+0000 7f558fd54640 10 monclient: init 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 10 monclient(hunting): picked mon.noname-c con 0x7f55880b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 10 monclient(hunting): picked mon.noname-a con 0x7f55880b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 10 monclient(hunting): picked mon.noname-b con 0x7f558814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880af940 con 0x7f55880b1730 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880b0a70 con 0x7f55880b4d60 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55880b0d50 con 0x7f558814be90 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:56.217 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.216+0000 7f558fd54640 10 monclient(hunting): authenticate will time out at 5131.480578s 2024-03-25T19:45:56.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.226+0000 7f558e2f1640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f558814be90 0x7f558814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:38680/0 (socket says 172.21.15.50:38680) 2024-03-25T19:45:56.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.226+0000 7f558e2f1640 1 -- 172.21.15.50:0/509220540 learned_addr learned my addr 172.21.15.50:0/509220540 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:56.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.239+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000459122 0 0) 0x7f55880b0a70 con 0x7f55880b4d60 2024-03-25T19:45:56.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.239+0000 7f558caee640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.239+0000 7f558caee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.239+0000 7f558caee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.239+0000 7f558caee640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:45:56.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.239+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5574003520 con 0x7f55880b4d60 2024-03-25T19:45:56.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.241+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499950020 0 0) 0x7f55880b0d50 con 0x7f558814be90 2024-03-25T19:45:56.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.241+0000 7f558caee640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.241+0000 7f558caee640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.241+0000 7f558caee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.241+0000 7f558caee640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:45:56.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.241+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55880b0a70 con 0x7f558814be90 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945114846 0 0) 0x7f5574003520 con 0x7f55880b4d60 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 >> v1:172.21.15.50:6790/0 conn(0x7f558814be90 legacy=0x7f558814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 >> v1:172.21.15.50:6789/0 conn(0x7f55880b1730 legacy=0x7f55880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 10 monclient: _finish_hunting 0 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 1 monclient: found mon.noname-c 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880b1030 con 0x7f55880b4d60 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 10 monclient: _finish_auth 0 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 10 monclient: _check_auth_tickets 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:56.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558fd54640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5584001690 con 0x7f55880b4d60 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558fd54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558caee640 10 monclient: dump: 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:56.260 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:56.261 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:56.261 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.259+0000 7f558fd54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:56.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55840017f0 con 0x7f55880b4d60 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558caee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558caee640 1 -- 172.21.15.50:0/509220540 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5584001c00 con 0x7f55880b4d60 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558fd54640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558fd54640 4 set_mon_vals no callback set 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558caee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558caee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.270+0000 7f558caee640 10 monclient: dump: 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.271 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 10 monclient: shutdown 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 1 -- 172.21.15.50:0/509220540 >> v1:172.21.15.76:6789/0 conn(0x7f55880b4d60 legacy=0x7f558813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 1 -- 172.21.15.50:0/509220540 shutdown_connections 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 1 -- 172.21.15.50:0/509220540 >> 172.21.15.50:0/509220540 conn(0x7f55880617a0 msgr2=0x7f5588061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 1 -- 172.21.15.50:0/509220540 shutdown_connections 2024-03-25T19:45:56.272 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 1 -- 172.21.15.50:0/509220540 wait complete. 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 10 monclient: build_initial_monmap 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.271+0000 7f558fd54640 10 monclient: monmap: 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:45:56.274 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.272+0000 7f558fd54640 1 librados: starting msgr at 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.272+0000 7f558fd54640 1 librados: starting objecter 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.272+0000 7f558fd54640 1 Processor -- start 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.272+0000 7f558fd54640 1 -- start start 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.272+0000 7f558fd54640 1 librados: setting wanted keys 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.272+0000 7f558fd54640 1 librados: calling monclient init 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.272+0000 7f558fd54640 10 monclient: init 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 10 monclient(hunting): picked mon.noname-b con 0x7f55880b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 10 monclient(hunting): picked mon.noname-a con 0x7f55880b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 10 monclient(hunting): picked mon.noname-c con 0x7f558814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5588137070 con 0x7f55880b4d60 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55881375c0 con 0x7f55880b1730 2024-03-25T19:45:56.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55881378a0 con 0x7f558814be90 2024-03-25T19:45:56.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:56.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.273+0000 7f558fd54640 10 monclient(hunting): authenticate will time out at 5131.537127s 2024-03-25T19:45:56.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.277+0000 7f558e2f1640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f558814be90 0x7f55881342f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:38704/0 (socket says 172.21.15.50:38704) 2024-03-25T19:45:56.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.277+0000 7f558e2f1640 1 -- 172.21.15.50:0/3011603623 learned_addr learned my addr 172.21.15.50:0/3011603623 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:45:56.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.278+0000 7f557e7fc640 10 client.?.objecter ms_handle_connect 0x7f558814be90 2024-03-25T19:45:56.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.278+0000 7f557e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.279+0000 7f557e7fc640 10 client.?.objecter ms_handle_connect 0x7f55880b1730 2024-03-25T19:45:56.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.279+0000 7f557e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.292+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731866058 0 0) 0x7f55881378a0 con 0x7f558814be90 2024-03-25T19:45:56.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.292+0000 7f557e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.292+0000 7f557e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.292+0000 7f557e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.292+0000 7f557e7fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.292+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f556c004eb0 con 0x7f558814be90 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.293+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151505700 0 0) 0x7f55881375c0 con 0x7f55880b1730 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.293+0000 7f557e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.293+0000 7f557e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.293+0000 7f557e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.293+0000 7f557e7fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:45:56.293 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.293+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55881378a0 con 0x7f55880b1730 2024-03-25T19:45:56.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.299+0000 7f557e7fc640 10 client.?.objecter ms_handle_connect 0x7f55880b4d60 2024-03-25T19:45:56.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.299+0000 7f557e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:56.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.310+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253623529 0 0) 0x7f5588137070 con 0x7f55880b4d60 2024-03-25T19:45:56.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.310+0000 7f557e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:56.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.310+0000 7f557e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:56.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.310+0000 7f557e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:56.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.310+0000 7f557e7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:45:56.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.310+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55881375c0 con 0x7f55880b4d60 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.314+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3997574672 0 0) 0x7f556c004eb0 con 0x7f558814be90 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.314+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5588137070 con 0x7f558814be90 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.314+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5580001c10 con 0x7f558814be90 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.314+0000 7f557e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.314+0000 7f557e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.314+0000 7f557e7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.314 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.315 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.315 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:56.315 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:56.315 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:56.315 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:56.315 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.314+0000 7f557e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.319+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3089935660 0 0) 0x7f55881378a0 con 0x7f55880b1730 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.319+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f556c004eb0 con 0x7f55880b1730 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.319+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5584001b00 con 0x7f55880b1730 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.319+0000 7f557e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.319+0000 7f557e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.319+0000 7f557e7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:56.319 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.320 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.320 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.320 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:56.320 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:56.320 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:56.320 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:56.320 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.322+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1325250805 0 0) 0x7f55881375c0 con 0x7f55880b4d60 2024-03-25T19:45:56.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.322+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55881378a0 con 0x7f55880b4d60 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.322+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5578001830 con 0x7f55880b4d60 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.322+0000 7f557e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.322+0000 7f557e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.322+0000 7f557e7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:56.322 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3423524945 0 0) 0x7f5588137070 con 0x7f558814be90 2024-03-25T19:45:56.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 >> v1:172.21.15.76:6789/0 conn(0x7f55880b1730 legacy=0x7f5588137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 >> v1:172.21.15.50:6789/0 conn(0x7f55880b4d60 legacy=0x7f5588133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:56.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 1 monclient: found mon.c 2024-03-25T19:45:56.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588134a00 con 0x7f558814be90 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f557e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 5 monclient: authenticate success, global_id 4439 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 10 monclient: _renew_subs 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55881372e0 con 0x7f558814be90 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 10 monclient: _renew_subs 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55881365f0 con 0x7f558814be90 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.323+0000 7f558fd54640 1 librados: init done 2024-03-25T19:45:56.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.324+0000 7f558fd54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.324+0000 7f558fd54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.324+0000 7f558fd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:56.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.324+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55880627b0 con 0x7f558814be90 2024-03-25T19:45:56.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.687+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5580004b60 con 0x7f558814be90 2024-03-25T19:45:56.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.687+0000 7f557e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:56.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.687+0000 7f557ffff640 4 set_mon_vals no callback set 2024-03-25T19:45:56.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.687+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5580004e70 con 0x7f558814be90 2024-03-25T19:45:56.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.687+0000 7f557e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:56.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.687+0000 7f557e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:56.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.687+0000 7f557e7fc640 10 monclient: dump: 2024-03-25T19:45:56.688 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:45:56.689 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:45:56.689 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.689 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:45:56.689 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:56.689 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:45:56.689 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:45:56.690 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:45:56.690 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:45:56.690 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:45:56.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.688+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f558001ae40 con 0x7f558814be90 2024-03-25T19:45:56.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.688+0000 7f558daf0640 10 monclient: get_auth_request con 0x7f556c06e0b0 auth_method 0 2024-03-25T19:45:56.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.689+0000 7f557e7fc640 10 client.4439.objecter ms_handle_connect 0x7f556c06e0b0 2024-03-25T19:45:56.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.694+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f5580054a60 con 0x7f558814be90 2024-03-25T19:45:56.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.694+0000 7f557e7fc640 10 client.4439.objecter ms_dispatch 0x7f5588000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:56.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.694+0000 7f557e7fc640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:56.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.694+0000 7f557e7fc640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:56.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.695+0000 7f557e7fc640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-25T19:45:56.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.695+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55800503a0 con 0x7f558814be90 2024-03-25T19:45:56.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.695+0000 7f557e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:56.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.695+0000 7f557e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:56.934 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.934+0000 7f558fd54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:56.934 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.934+0000 7f558fd54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:56.934 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.934+0000 7f558fd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:45:56.934 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:56.934+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f558813b8a0 con 0x7f558814be90 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.316+0000 7f557e7fc640 1 -- 172.21.15.50:0/3011603623 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f55800524a0 con 0x7f558814be90 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.316+0000 7f557e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.316+0000 7f557e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:45:57.316 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:57.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 10 librados: watch_flush enter 2024-03-25T19:45:57.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 10 librados: watch_flush exit 2024-03-25T19:45:57.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-25T19:45:57.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 10 client.4439.objecter successfully canceled tick 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 >> v1:172.21.15.76:6800/2572556811 conn(0x7f556c06e0b0 legacy=0x7f556c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 10 monclient: shutdown 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 >> v1:172.21.15.50:6790/0 conn(0x7f558814be90 legacy=0x7f55881342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 shutdown_connections 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 >> 172.21.15.50:0/3011603623 conn(0x7f55880617a0 msgr2=0x7f5588061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 shutdown_connections 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.319+0000 7f558fd54640 1 -- 172.21.15.50:0/3011603623 wait complete. 2024-03-25T19:45:57.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:45:57.320+0000 7f558fd54640 1 librados: shutdown 2024-03-25T19:45:57.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:04.139 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:04.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.254+0000 7f9050f10640 10 monclient: get_monmap_and_config 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.254+0000 7f9050f10640 10 monclient: build_initial_monmap 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.254+0000 7f9050f10640 10 monclient: monmap: 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:04.255150+0000 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:04.255150+0000 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:04.254 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.255+0000 7f9050f10640 1 Processor -- start 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.255+0000 7f9050f10640 1 -- start start 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.255+0000 7f9050f10640 10 monclient: init 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 10 monclient(hunting): picked mon.noname-c con 0x7f904c0a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 10 monclient(hunting): picked mon.noname-b con 0x7f904c13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 10 monclient(hunting): picked mon.noname-a con 0x7f904c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:04.256 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c142520 con 0x7f904c1430c0 2024-03-25T19:46:04.257 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0b28a0 con 0x7f904c0a98f0 2024-03-25T19:46:04.257 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0b2a30 con 0x7f904c13f710 2024-03-25T19:46:04.257 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:04.257 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.256+0000 7f9050f10640 10 monclient(hunting): authenticate will time out at 5139.520050s 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.265+0000 7f904a59c640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f904c0a98f0 0x7f904c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:40596/0 (socket says 172.21.15.50:40596) 2024-03-25T19:46:04.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.265+0000 7f904a59c640 1 -- 172.21.15.50:0/1466038536 learned_addr learned my addr 172.21.15.50:0/1466038536 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:04.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.279+0000 7f9049d9b640 1 -- 172.21.15.50:0/1466038536 >> v1:172.21.15.50:6790/0 conn(0x7f904c13f710 legacy=0x7f904c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T19:46:04.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.279+0000 7f9049d9b640 1 -- 172.21.15.50:0/1466038536 >> v1:172.21.15.50:6790/0 conn(0x7f904c13f710 legacy=0x7f904c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:46:04.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.279+0000 7f9049d9b640 1 --1- 172.21.15.50:0/1466038536 >> v1:172.21.15.50:6790/0 conn(0x7f904c13f710 0x7f904c1429b0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:46:04.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.285+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794513784 0 0) 0x7f904c0b28a0 con 0x7f904c0a98f0 2024-03-25T19:46:04.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.285+0000 7f904959a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.285+0000 7f904959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.285+0000 7f904959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.285+0000 7f904959a640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:46:04.284 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.285+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9034003520 con 0x7f904c0a98f0 2024-03-25T19:46:04.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.301+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965353858 0 0) 0x7f904c142520 con 0x7f904c1430c0 2024-03-25T19:46:04.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.301+0000 7f904959a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.301+0000 7f904959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.301+0000 7f904959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.301+0000 7f904959a640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:46:04.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.301+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c0b28a0 con 0x7f904c1430c0 2024-03-25T19:46:04.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3783501659 0 0) 0x7f904c0b28a0 con 0x7f904c1430c0 2024-03-25T19:46:04.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 >> v1:172.21.15.50:6790/0 conn(0x7f904c13f710 legacy=0x7f904c1429b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:46:04.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 >> v1:172.21.15.76:6789/0 conn(0x7f904c0a98f0 legacy=0x7f904c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 10 monclient: _finish_hunting 0 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 1 monclient: found mon.noname-a 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c0b44c0 con 0x7f904c1430c0 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 10 monclient: _finish_auth 0 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 10 monclient: _check_auth_tickets 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f90440019f0 con 0x7f904c1430c0 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f904959a640 10 monclient: dump: 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:04.315 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f9050f10640 5 monclient: authenticate success, global_id 4464 2024-03-25T19:46:04.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.314+0000 7f9050f10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:04.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9044001b50 con 0x7f904c1430c0 2024-03-25T19:46:04.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f904959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f904959a640 1 -- 172.21.15.50:0/1466038536 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9044001f60 con 0x7f904c1430c0 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f904959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f904959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f904959a640 10 monclient: dump: 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:04.319 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f9050f10640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:04.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.319+0000 7f9050f10640 4 set_mon_vals no callback set 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.320+0000 7f9050f10640 10 monclient: shutdown 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.320+0000 7f9050f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.320+0000 7f9050f10640 1 -- 172.21.15.50:0/1466038536 >> v1:172.21.15.50:6789/0 conn(0x7f904c1430c0 legacy=0x7f904c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.320+0000 7f9050f10640 1 -- 172.21.15.50:0/1466038536 shutdown_connections 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.320+0000 7f9050f10640 1 -- 172.21.15.50:0/1466038536 >> 172.21.15.50:0/1466038536 conn(0x7f904c0b7ea0 msgr2=0x7f904c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.320+0000 7f9050f10640 1 -- 172.21.15.50:0/1466038536 shutdown_connections 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.320+0000 7f9050f10640 1 -- 172.21.15.50:0/1466038536 wait complete. 2024-03-25T19:46:04.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.321+0000 7f9050f10640 10 monclient: build_initial_monmap 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.321+0000 7f9050f10640 10 monclient: monmap: 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.321+0000 7f9050f10640 1 librados: starting msgr at 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.321+0000 7f9050f10640 1 librados: starting objecter 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 1 Processor -- start 2024-03-25T19:46:04.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 1 -- start start 2024-03-25T19:46:04.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 1 librados: setting wanted keys 2024-03-25T19:46:04.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 1 librados: calling monclient init 2024-03-25T19:46:04.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient: init 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient(hunting): picked mon.noname-c con 0x7f904c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient(hunting): picked mon.noname-b con 0x7f904c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient(hunting): picked mon.noname-a con 0x7f904c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0ab9c0 con 0x7f904c1430c0 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c139950 con 0x7f904c13f710 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c1383e0 con 0x7f904c0a98f0 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:04.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.322+0000 7f9050f10640 10 monclient(hunting): authenticate will time out at 5139.586534s 2024-03-25T19:46:04.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.337+0000 7f904ad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f904c1430c0 0x7f904c1e1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:37820/0 (socket says 172.21.15.50:37820) 2024-03-25T19:46:04.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.337+0000 7f904ad9d640 1 -- 172.21.15.50:0/2314555424 learned_addr learned my addr 172.21.15.50:0/2314555424 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:04.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.338+0000 7f903affd640 10 client.?.objecter ms_handle_connect 0x7f904c1430c0 2024-03-25T19:46:04.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.338+0000 7f903affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:04.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.339+0000 7f903affd640 10 client.?.objecter ms_handle_connect 0x7f904c13f710 2024-03-25T19:46:04.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.339+0000 7f903affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:04.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.343+0000 7f903affd640 10 client.?.objecter ms_handle_connect 0x7f904c0a98f0 2024-03-25T19:46:04.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.343+0000 7f903affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:04.348 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.349+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897446858 0 0) 0x7f904c0ab9c0 con 0x7f904c1430c0 2024-03-25T19:46:04.348 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.349+0000 7f903affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.348 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.349+0000 7f903affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.349+0000 7f903affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.349+0000 7f903affd640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:46:04.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.349+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902c004eb0 con 0x7f904c1430c0 2024-03-25T19:46:04.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.350+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454107016 0 0) 0x7f904c139950 con 0x7f904c13f710 2024-03-25T19:46:04.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.350+0000 7f903affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.350+0000 7f903affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.350+0000 7f903affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.350+0000 7f903affd640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:46:04.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.350+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c0ab9c0 con 0x7f904c13f710 2024-03-25T19:46:04.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.370+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964640657 0 0) 0x7f904c1383e0 con 0x7f904c0a98f0 2024-03-25T19:46:04.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.370+0000 7f903affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:04.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.370+0000 7f903affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:04.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.370+0000 7f903affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:04.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.370+0000 7f903affd640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:46:04.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.370+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f904c139950 con 0x7f904c0a98f0 2024-03-25T19:46:04.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.381+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888821413 0 0) 0x7f904c0ab9c0 con 0x7f904c13f710 2024-03-25T19:46:04.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.381+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f904c1383e0 con 0x7f904c13f710 2024-03-25T19:46:04.381 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.381+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9040003fd0 con 0x7f904c13f710 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.381+0000 7f903affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.381+0000 7f903affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.381+0000 7f903affd640 10 monclient(hunting): dump: 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.381+0000 7f903affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.382+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4211640215 0 0) 0x7f902c004eb0 con 0x7f904c1430c0 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.382+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f904c0ab9c0 con 0x7f904c1430c0 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.382+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f90440019b0 con 0x7f904c1430c0 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.382+0000 7f903affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.382+0000 7f903affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.382+0000 7f903affd640 10 monclient(hunting): dump: 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:04.383 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:04.384 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.388+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 655470369 0 0) 0x7f904c139950 con 0x7f904c0a98f0 2024-03-25T19:46:04.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.388+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f902c004eb0 con 0x7f904c0a98f0 2024-03-25T19:46:04.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.388+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f903c0017b0 con 0x7f904c0a98f0 2024-03-25T19:46:04.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.388+0000 7f903affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.388+0000 7f903affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.388+0000 7f903affd640 10 monclient(hunting): dump: 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:04.388 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4006253681 0 0) 0x7f904c1383e0 con 0x7f904c13f710 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 >> v1:172.21.15.50:6790/0 conn(0x7f904c0a98f0 legacy=0x7f904c135f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 >> v1:172.21.15.50:6789/0 conn(0x7f904c1430c0 legacy=0x7f904c1e1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 1 monclient: found mon.b 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c138b60 con 0x7f904c13f710 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 10 monclient: _finish_auth 0 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f903affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:04.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f9050f10640 5 monclient: authenticate success, global_id 4447 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.401+0000 7f9050f10640 10 monclient: _renew_subs 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.402+0000 7f9050f10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.402+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c138e10 con 0x7f904c13f710 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.402+0000 7f9050f10640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.402+0000 7f9050f10640 10 monclient: _renew_subs 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.402+0000 7f9050f10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.402+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c1db270 con 0x7f904c13f710 2024-03-25T19:46:04.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.402+0000 7f9050f10640 1 librados: init done 2024-03-25T19:46:04.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.403+0000 7f9050f10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:04.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.403+0000 7f9050f10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:04.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.403+0000 7f9050f10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:04.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.403+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c138570 con 0x7f904c13f710 2024-03-25T19:46:04.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.702+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9040000e80 con 0x7f904c13f710 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.702+0000 7f903affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.702+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f90400044e0 con 0x7f904c13f710 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.702+0000 7f903affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.702+0000 7f903affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.702+0000 7f903affd640 10 monclient: dump: 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:04.703 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:04.704 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.702+0000 7f9048d99640 4 set_mon_vals no callback set 2024-03-25T19:46:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.703+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f90400012d0 con 0x7f904c13f710 2024-03-25T19:46:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.704+0000 7f904a59c640 10 monclient: get_auth_request con 0x7f902c06e0b0 auth_method 0 2024-03-25T19:46:04.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.704+0000 7f903affd640 10 client.4447.objecter ms_handle_connect 0x7f902c06e0b0 2024-03-25T19:46:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.709+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f9040054750 con 0x7f904c13f710 2024-03-25T19:46:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.709+0000 7f903affd640 10 client.4447.objecter ms_dispatch 0x7f904c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:04.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.710+0000 7f903affd640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:04.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.710+0000 7f903affd640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:04.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.710+0000 7f903affd640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-25T19:46:04.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.710+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9040050090 con 0x7f904c13f710 2024-03-25T19:46:04.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.710+0000 7f903affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:04.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.710+0000 7f903affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:04.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.942+0000 7f9050f10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.942+0000 7f9050f10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.942+0000 7f9050f10640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:04.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.942+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f904c138990 con 0x7f904c13f710 2024-03-25T19:46:04.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.976+0000 7f903affd640 1 -- 172.21.15.50:0/2314555424 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f9040052190 con 0x7f904c13f710 2024-03-25T19:46:04.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.976+0000 7f903affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:04.976 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.976+0000 7f903affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:04.976 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:04.976 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:04.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 10 librados: watch_flush enter 2024-03-25T19:46:04.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 10 librados: watch_flush exit 2024-03-25T19:46:04.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-25T19:46:04.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 10 client.4447.objecter successfully canceled tick 2024-03-25T19:46:04.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 >> v1:172.21.15.76:6800/2572556811 conn(0x7f902c06e0b0 legacy=0x7f902c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 10 monclient: shutdown 2024-03-25T19:46:04.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:04.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 >> v1:172.21.15.76:6789/0 conn(0x7f904c13f710 legacy=0x7f904c1da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:04.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 shutdown_connections 2024-03-25T19:46:04.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 >> 172.21.15.50:0/2314555424 conn(0x7f904c0b7ea0 msgr2=0x7f904c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:04.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 shutdown_connections 2024-03-25T19:46:04.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 1 -- 172.21.15.50:0/2314555424 wait complete. 2024-03-25T19:46:04.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:04.978+0000 7f9050f10640 1 librados: shutdown 2024-03-25T19:46:04.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:11.796 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.909+0000 7f92d457b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.909+0000 7f92d457b640 10 monclient: build_initial_monmap 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.909+0000 7f92d457b640 10 monclient: monmap: 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:11.910561+0000 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:11.910561+0000 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:11.909 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:11.910 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:11.910 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:11.910 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.910+0000 7f92d457b640 1 Processor -- start 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.910+0000 7f92d457b640 1 -- start start 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.910+0000 7f92d457b640 10 monclient: init 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 10 monclient(hunting): picked mon.noname-a con 0x7f92cc0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 10 monclient(hunting): picked mon.noname-b con 0x7f92cc13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 10 monclient(hunting): picked mon.noname-c con 0x7f92cc1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc142520 con 0x7f92cc0a98f0 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0b28a0 con 0x7f92cc1430c0 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0b2a30 con 0x7f92cc13f710 2024-03-25T19:46:11.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:11.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.911+0000 7f92d457b640 10 monclient(hunting): authenticate will time out at 5147.174981s 2024-03-25T19:46:11.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.919+0000 7f92d1b16640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f92cc13f710 0x7f92cc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:54698/0 (socket says 172.21.15.50:54698) 2024-03-25T19:46:11.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.919+0000 7f92d1b16640 1 -- 172.21.15.50:0/66375925 learned_addr learned my addr 172.21.15.50:0/66375925 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549315371 0 0) 0x7f92cc0b28a0 con 0x7f92cc1430c0 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b8003520 con 0x7f92cc1430c0 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150314737 0 0) 0x7f92cc0b2a30 con 0x7f92cc13f710 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:46:11.933 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.932+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc0b28a0 con 0x7f92cc13f710 2024-03-25T19:46:11.943 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.944+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492855228 0 0) 0x7f92cc142520 con 0x7f92cc0a98f0 2024-03-25T19:46:11.944 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.944+0000 7f92d1315640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.944 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.944+0000 7f92d1315640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.944 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.944+0000 7f92d1315640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.944 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.944+0000 7f92d1315640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:46:11.944 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.944+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc0b2a30 con 0x7f92cc0a98f0 2024-03-25T19:46:11.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1909698152 0 0) 0x7f92b8003520 con 0x7f92cc1430c0 2024-03-25T19:46:11.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 >> v1:172.21.15.50:6790/0 conn(0x7f92cc13f710 legacy=0x7f92cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 >> v1:172.21.15.50:6789/0 conn(0x7f92cc0a98f0 legacy=0x7f92cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 10 monclient: _finish_hunting 0 2024-03-25T19:46:11.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 1 monclient: found mon.noname-c 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0b44c0 con 0x7f92cc1430c0 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 10 monclient: _finish_auth 0 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 10 monclient: _check_auth_tickets 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f92c4001a10 con 0x7f92cc1430c0 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d1315640 10 monclient: dump: 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:11.950 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d457b640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:46:11.951 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.949+0000 7f92d457b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92c4001b70 con 0x7f92cc1430c0 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d1315640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d1315640 1 -- 172.21.15.50:0/66375925 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f92c4001f80 con 0x7f92cc1430c0 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d1315640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d1315640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d1315640 10 monclient: dump: 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:11.964 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:11.965 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:11.965 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:11.965 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:11.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d457b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:11.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.964+0000 7f92d457b640 4 set_mon_vals no callback set 2024-03-25T19:46:11.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.965+0000 7f92d457b640 10 monclient: shutdown 2024-03-25T19:46:11.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.965+0000 7f92d457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:11.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.965+0000 7f92d457b640 1 -- 172.21.15.50:0/66375925 >> v1:172.21.15.76:6789/0 conn(0x7f92cc1430c0 legacy=0x7f92cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.965+0000 7f92d457b640 1 -- 172.21.15.50:0/66375925 shutdown_connections 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.965+0000 7f92d457b640 1 -- 172.21.15.50:0/66375925 >> 172.21.15.50:0/66375925 conn(0x7f92cc0b7ea0 msgr2=0x7f92cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.965+0000 7f92d457b640 1 -- 172.21.15.50:0/66375925 shutdown_connections 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.965+0000 7f92d457b640 1 -- 172.21.15.50:0/66375925 wait complete. 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 10 monclient: build_initial_monmap 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 10 monclient: monmap: 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:11.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 1 librados: starting msgr at 2024-03-25T19:46:11.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 1 librados: starting objecter 2024-03-25T19:46:11.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 1 Processor -- start 2024-03-25T19:46:11.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 1 -- start start 2024-03-25T19:46:11.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 1 librados: setting wanted keys 2024-03-25T19:46:11.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 1 librados: calling monclient init 2024-03-25T19:46:11.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.966+0000 7f92d457b640 10 monclient: init 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 10 monclient(hunting): picked mon.noname-a con 0x7f92cc0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 10 monclient(hunting): picked mon.noname-b con 0x7f92cc13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 10 monclient(hunting): picked mon.noname-c con 0x7f92cc1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc1daca0 con 0x7f92cc0a98f0 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc138ab0 con 0x7f92cc13f710 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0aba90 con 0x7f92cc1430c0 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:11.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.967+0000 7f92d457b640 10 monclient(hunting): authenticate will time out at 5147.231590s 2024-03-25T19:46:11.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.980+0000 7f92d2317640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f92cc0a98f0 0x7f92cc1383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:33466/0 (socket says 172.21.15.50:33466) 2024-03-25T19:46:11.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.980+0000 7f92d2317640 1 -- 172.21.15.50:0/3569088682 learned_addr learned my addr 172.21.15.50:0/3569088682 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:11.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.981+0000 7f92c2ffd640 10 client.?.objecter ms_handle_connect 0x7f92cc0a98f0 2024-03-25T19:46:11.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.981+0000 7f92c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:11.982 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.982+0000 7f92c2ffd640 10 client.?.objecter ms_handle_connect 0x7f92cc13f710 2024-03-25T19:46:11.982 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.982+0000 7f92c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:11.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.991+0000 7f92c2ffd640 10 client.?.objecter ms_handle_connect 0x7f92cc1430c0 2024-03-25T19:46:11.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.991+0000 7f92c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.994+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836346220 0 0) 0x7f92cc1daca0 con 0x7f92cc0a98f0 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.994+0000 7f92c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.994+0000 7f92c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:11.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.994+0000 7f92c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.994+0000 7f92c2ffd640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:46:11.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:11.994+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b0004eb0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.012+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 236811365 0 0) 0x7f92b0004eb0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.012+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92cc1daca0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.012+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f92c8001a10 con 0x7f92cc0a98f0 2024-03-25T19:46:12.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.012+0000 7f92c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.012+0000 7f92c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:12.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.012+0000 7f92c2ffd640 10 monclient(hunting): dump: 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:12.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.012+0000 7f92c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1691689863 0 0) 0x7f92cc1daca0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 >> v1:172.21.15.50:6790/0 conn(0x7f92cc1430c0 legacy=0x7f92cc1d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 >> v1:172.21.15.76:6789/0 conn(0x7f92cc13f710 legacy=0x7f92cc1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 1 monclient: found mon.a 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc1d41e0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:12.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 5 monclient: authenticate success, global_id 4473 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 10 monclient: _renew_subs 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc1db840 con 0x7f92cc0a98f0 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 10 monclient: _renew_subs 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc138f00 con 0x7f92cc0a98f0 2024-03-25T19:46:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.090+0000 7f92d457b640 1 librados: init done 2024-03-25T19:46:12.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.091+0000 7f92d457b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.091+0000 7f92d457b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.091+0000 7f92d457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:12.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.091+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc1dba80 con 0x7f92cc0a98f0 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.401+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92c8001cb0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.401+0000 7f92c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.401+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f92c8003a60 con 0x7f92cc0a98f0 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.401+0000 7f92c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.401+0000 7f92c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.401+0000 7f92c2ffd640 10 monclient: dump: 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:12.401 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.401+0000 7f92d0b14640 4 set_mon_vals no callback set 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.402+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f92c8003c80 con 0x7f92cc0a98f0 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.402+0000 7f92d1b16640 10 monclient: get_auth_request con 0x7f92b006dd00 auth_method 0 2024-03-25T19:46:12.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.403+0000 7f92c2ffd640 10 client.4473.objecter ms_handle_connect 0x7f92b006dd00 2024-03-25T19:46:12.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f92c8054890 con 0x7f92cc0a98f0 2024-03-25T19:46:12.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 10 client.4473.objecter ms_dispatch 0x7f92cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:12.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:12.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:12.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T19:46:12.409 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92c80503a0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.410 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:12.410 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.409+0000 7f92c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:12.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.651+0000 7f92d457b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.651+0000 7f92d457b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.651+0000 7f92d457b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:12.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.651+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92cc0627b0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.674+0000 7f92c2ffd640 1 -- 172.21.15.50:0/3569088682 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f92c80522d0 con 0x7f92cc0a98f0 2024-03-25T19:46:12.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.674+0000 7f92c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:12.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.674+0000 7f92c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:12.674 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:12.674 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:12.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 10 librados: watch_flush enter 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 10 librados: watch_flush exit 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 10 client.4473.objecter successfully canceled tick 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 >> v1:172.21.15.76:6800/2572556811 conn(0x7f92b006dd00 legacy=0x7f92b00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 10 monclient: shutdown 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 >> v1:172.21.15.50:6789/0 conn(0x7f92cc0a98f0 legacy=0x7f92cc1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 shutdown_connections 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 >> 172.21.15.50:0/3569088682 conn(0x7f92cc0b7ea0 msgr2=0x7f92cc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.676+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 shutdown_connections 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.677+0000 7f92d457b640 1 -- 172.21.15.50:0/3569088682 wait complete. 2024-03-25T19:46:12.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:12.677+0000 7f92d457b640 1 librados: shutdown 2024-03-25T19:46:12.688 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:15.036 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:05:35.542 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.046 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:05:35.552 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.074 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:05:35.580 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.098 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:05:35.568 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.105 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:05:35.576 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.116 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:05:35.622 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.127 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:05:35.633 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.133 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:05:35.604 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:15.157 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:05:35.662 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:19.494 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:19.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.607+0000 7fd2204e6640 10 monclient: get_monmap_and_config 2024-03-25T19:46:19.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.607+0000 7fd2204e6640 10 monclient: build_initial_monmap 2024-03-25T19:46:19.608 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.607+0000 7fd2204e6640 10 monclient: monmap: 2024-03-25T19:46:19.608 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:19.608 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:19.609722+0000 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:19.609722+0000 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:19.609 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:19.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.608+0000 7fd2204e6640 1 Processor -- start 2024-03-25T19:46:19.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.608+0000 7fd2204e6640 1 -- start start 2024-03-25T19:46:19.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.608+0000 7fd2204e6640 10 monclient: init 2024-03-25T19:46:19.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:19.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2180b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd2180b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd21814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180af940 con 0x7fd2180b4d60 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180b0a70 con 0x7fd21814be90 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2180b0d50 con 0x7fd2180b1730 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:19.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.609+0000 7fd2204e6640 10 monclient(hunting): authenticate will time out at 5154.874138s 2024-03-25T19:46:19.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.617+0000 7fd21da81640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fd2180b1730 0x7fd2180b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:58624/0 (socket says 172.21.15.50:58624) 2024-03-25T19:46:19.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.618+0000 7fd21da81640 1 -- 172.21.15.50:0/2039845200 learned_addr learned my addr 172.21.15.50:0/2039845200 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:19.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.641+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062461440 0 0) 0x7fd2180b0d50 con 0x7fd2180b1730 2024-03-25T19:46:19.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.641+0000 7fd21d280640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.641+0000 7fd21d280640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.641+0000 7fd21d280640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.641+0000 7fd21d280640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:46:19.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.641+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd204003520 con 0x7fd2180b1730 2024-03-25T19:46:19.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.642+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301476296 0 0) 0x7fd2180af940 con 0x7fd2180b4d60 2024-03-25T19:46:19.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.642+0000 7fd21d280640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.642+0000 7fd21d280640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.642+0000 7fd21d280640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.642+0000 7fd21d280640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:46:19.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.642+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2180b0d50 con 0x7fd2180b4d60 2024-03-25T19:46:19.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.649+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368599031 0 0) 0x7fd2180b0a70 con 0x7fd21814be90 2024-03-25T19:46:19.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.649+0000 7fd21d280640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.649+0000 7fd21d280640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.649+0000 7fd21d280640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.649+0000 7fd21d280640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:46:19.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.649+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2180af940 con 0x7fd21814be90 2024-03-25T19:46:19.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567316867 0 0) 0x7fd204003520 con 0x7fd2180b1730 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 >> v1:172.21.15.76:6789/0 conn(0x7fd21814be90 legacy=0x7fd21814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 >> v1:172.21.15.50:6789/0 conn(0x7fd2180b4d60 legacy=0x7fd21813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 10 monclient: _finish_hunting 0 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 1 monclient: found mon.noname-b 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.650+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2180b1030 con 0x7fd2180b1730 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd21d280640 10 monclient: _finish_auth 0 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd21d280640 10 monclient: _check_auth_tickets 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd21d280640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fd208001690 con 0x7fd2180b1730 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd21d280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd21d280640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:19.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd21d280640 10 monclient: dump: 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd2204e6640 5 monclient: authenticate success, global_id 4451 2024-03-25T19:46:19.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.651+0000 7fd2204e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2080017f0 con 0x7fd2180b1730 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd21d280640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd21d280640 1 -- 172.21.15.50:0/2039845200 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fd208001c00 con 0x7fd2180b1730 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd2204e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd2204e6640 4 set_mon_vals no callback set 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd21d280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd21d280640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.663+0000 7fd21d280640 10 monclient: dump: 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.664 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:19.665 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:19.665 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:19.665 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:19.665 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.664+0000 7fd2204e6640 10 monclient: shutdown 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.664+0000 7fd2204e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.664+0000 7fd2204e6640 1 -- 172.21.15.50:0/2039845200 >> v1:172.21.15.50:6790/0 conn(0x7fd2180b1730 legacy=0x7fd2180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.664+0000 7fd2204e6640 1 -- 172.21.15.50:0/2039845200 shutdown_connections 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.664+0000 7fd2204e6640 1 -- 172.21.15.50:0/2039845200 >> 172.21.15.50:0/2039845200 conn(0x7fd2180617a0 msgr2=0x7fd218061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.664+0000 7fd2204e6640 1 -- 172.21.15.50:0/2039845200 shutdown_connections 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.664+0000 7fd2204e6640 1 -- 172.21.15.50:0/2039845200 wait complete. 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 10 monclient: build_initial_monmap 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 10 monclient: monmap: 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:19.667 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 1 librados: starting msgr at 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 1 librados: starting objecter 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 1 Processor -- start 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 1 -- start start 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 1 librados: setting wanted keys 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 1 librados: calling monclient init 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.665+0000 7fd2204e6640 10 monclient: init 2024-03-25T19:46:19.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd2180b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2180b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd21814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2181409c0 con 0x7fd2180b4d60 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd218140ee0 con 0x7fd2180b1730 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd218141070 con 0x7fd21814be90 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:19.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.666+0000 7fd2204e6640 10 monclient(hunting): authenticate will time out at 5154.931024s 2024-03-25T19:46:19.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.672+0000 7fd21e282640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd2180b1730 0x7fd218131ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:48094/0 (socket says 172.21.15.50:48094) 2024-03-25T19:46:19.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.672+0000 7fd21e282640 1 -- 172.21.15.50:0/515207444 learned_addr learned my addr 172.21.15.50:0/515207444 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:19.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.676+0000 7fd20effd640 10 client.?.objecter ms_handle_connect 0x7fd2180b1730 2024-03-25T19:46:19.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.676+0000 7fd20effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.686+0000 7fd20effd640 10 client.?.objecter ms_handle_connect 0x7fd2180b4d60 2024-03-25T19:46:19.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.686+0000 7fd20effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.688+0000 7fd20effd640 10 client.?.objecter ms_handle_connect 0x7fd21814be90 2024-03-25T19:46:19.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.688+0000 7fd20effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.689+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216774177 0 0) 0x7fd218140ee0 con 0x7fd2180b1730 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.689+0000 7fd20effd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.689+0000 7fd20effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.689+0000 7fd20effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.689+0000 7fd20effd640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:46:19.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.689+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1fc004eb0 con 0x7fd2180b1730 2024-03-25T19:46:19.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.707+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303906602 0 0) 0x7fd2181409c0 con 0x7fd2180b4d60 2024-03-25T19:46:19.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.707+0000 7fd20effd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.707+0000 7fd20effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.707+0000 7fd20effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.707+0000 7fd20effd640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:46:19.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.707+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd218140ee0 con 0x7fd2180b4d60 2024-03-25T19:46:19.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.709+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575991985 0 0) 0x7fd218141070 con 0x7fd21814be90 2024-03-25T19:46:19.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.709+0000 7fd20effd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:19.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.709+0000 7fd20effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:19.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.709+0000 7fd20effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:19.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.709+0000 7fd20effd640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:46:19.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.709+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2181409c0 con 0x7fd21814be90 2024-03-25T19:46:19.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.717+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3588914999 0 0) 0x7fd1fc004eb0 con 0x7fd2180b1730 2024-03-25T19:46:19.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.717+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd218141070 con 0x7fd2180b1730 2024-03-25T19:46:19.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.717+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fd214001a30 con 0x7fd2180b1730 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.717+0000 7fd20effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.717+0000 7fd20effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.717+0000 7fd20effd640 10 monclient(hunting): dump: 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:19.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.717+0000 7fd20effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:19.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.727+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483968464 0 0) 0x7fd2181409c0 con 0x7fd21814be90 2024-03-25T19:46:19.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.727+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1fc004eb0 con 0x7fd21814be90 2024-03-25T19:46:19.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.727+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fd210001bd0 con 0x7fd21814be90 2024-03-25T19:46:19.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.727+0000 7fd20effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.728+0000 7fd20effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:19.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.728+0000 7fd20effd640 10 monclient(hunting): dump: 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:19.729 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.739+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2618289041 0 0) 0x7fd218140ee0 con 0x7fd2180b4d60 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.739+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2181409c0 con 0x7fd2180b4d60 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.739+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fd208001b00 con 0x7fd2180b4d60 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.739+0000 7fd20effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.739+0000 7fd20effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:19.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.739+0000 7fd20effd640 10 monclient(hunting): dump: 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:19.741 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3709872016 0 0) 0x7fd218141070 con 0x7fd2180b1730 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 >> v1:172.21.15.50:6790/0 conn(0x7fd21814be90 legacy=0x7fd2181e2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 >> v1:172.21.15.50:6789/0 conn(0x7fd2180b4d60 legacy=0x7fd2181424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 1 monclient: found mon.b 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:19.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd218145560 con 0x7fd2180b1730 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 10 monclient: _finish_auth 0 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd20effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 10 monclient: _renew_subs 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2181325b0 con 0x7fd2180b1730 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 10 monclient: _renew_subs 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2181387b0 con 0x7fd2180b1730 2024-03-25T19:46:19.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.741+0000 7fd2204e6640 1 librados: init done 2024-03-25T19:46:19.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.742+0000 7fd2204e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:19.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.742+0000 7fd2204e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:19.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.742+0000 7fd2204e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:19.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:19.742+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd218140b70 con 0x7fd2180b1730 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.013+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd214001c70 con 0x7fd2180b1730 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.013+0000 7fd20effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.013+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fd2140038c0 con 0x7fd2180b1730 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.013+0000 7fd20effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.013+0000 7fd20effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.013+0000 7fd20effd640 10 monclient: dump: 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:20.014 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.013+0000 7fd21ca7f640 4 set_mon_vals no callback set 2024-03-25T19:46:20.015 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.014+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fd214003ae0 con 0x7fd2180b1730 2024-03-25T19:46:20.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.015+0000 7fd21da81640 10 monclient: get_auth_request con 0x7fd1fc06e0f0 auth_method 0 2024-03-25T19:46:20.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.015+0000 7fd20effd640 10 client.4459.objecter ms_handle_connect 0x7fd1fc06e0f0 2024-03-25T19:46:20.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.021+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fd2140546b0 con 0x7fd2180b1730 2024-03-25T19:46:20.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.021+0000 7fd20effd640 10 client.4459.objecter ms_dispatch 0x7fd218000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:20.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.021+0000 7fd20effd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:20.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.021+0000 7fd20effd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:20.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.021+0000 7fd20effd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T19:46:20.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.022+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2140501c0 con 0x7fd2180b1730 2024-03-25T19:46:20.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.023+0000 7fd20effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:20.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.023+0000 7fd20effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:20.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.270+0000 7fd2204e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:20.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.270+0000 7fd2204e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:20.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.270+0000 7fd2204e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:20.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.270+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2181421a0 con 0x7fd2180b1730 2024-03-25T19:46:20.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.298+0000 7fd20effd640 1 -- 172.21.15.50:0/515207444 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fd2140520f0 con 0x7fd2180b1730 2024-03-25T19:46:20.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.298+0000 7fd20effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:20.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.298+0000 7fd20effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:20.299 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:20.300 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 10 librados: watch_flush enter 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 10 librados: watch_flush exit 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 10 client.4459.objecter successfully canceled tick 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 >> v1:172.21.15.76:6800/2572556811 conn(0x7fd1fc06e0f0 legacy=0x7fd1fc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 10 monclient: shutdown 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:20.302 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 >> v1:172.21.15.76:6789/0 conn(0x7fd2180b1730 legacy=0x7fd218131ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:20.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 shutdown_connections 2024-03-25T19:46:20.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 >> 172.21.15.50:0/515207444 conn(0x7fd2180617a0 msgr2=0x7fd21813c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:20.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 shutdown_connections 2024-03-25T19:46:20.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 1 -- 172.21.15.50:0/515207444 wait complete. 2024-03-25T19:46:20.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:20.301+0000 7fd2204e6640 1 librados: shutdown 2024-03-25T19:46:20.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:27.120 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:27.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.236+0000 7f7ecf8c1640 10 monclient: get_monmap_and_config 2024-03-25T19:46:27.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.236+0000 7f7ecf8c1640 10 monclient: build_initial_monmap 2024-03-25T19:46:27.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.236+0000 7f7ecf8c1640 10 monclient: monmap: 2024-03-25T19:46:27.236 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:27.236 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:27.236 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:27.237552+0000 2024-03-25T19:46:27.237 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:27.237552+0000 2024-03-25T19:46:27.237 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:27.237 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:27.237 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:27.237 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:27.237 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:27.237 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.237+0000 7f7ecf8c1640 1 Processor -- start 2024-03-25T19:46:27.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.237+0000 7f7ecf8c1640 1 -- start start 2024-03-25T19:46:27.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.237+0000 7f7ecf8c1640 10 monclient: init 2024-03-25T19:46:27.238 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec80a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec81430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8142520 con 0x7f7ec80a98f0 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b28a0 con 0x7f7ec81430c0 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b2a30 con 0x7f7ec813f710 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:27.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.238+0000 7f7ecf8c1640 10 monclient(hunting): authenticate will time out at 5162.502240s 2024-03-25T19:46:27.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.240+0000 7f7ecce5c640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f7ec813f710 0x7f7ec81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:35018/0 (socket says 172.21.15.50:35018) 2024-03-25T19:46:27.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.240+0000 7f7ecce5c640 1 -- 172.21.15.50:0/34938774 learned_addr learned my addr 172.21.15.50:0/34938774 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:27.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.258+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497677518 0 0) 0x7f7ec80b2a30 con 0x7f7ec813f710 2024-03-25T19:46:27.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.258+0000 7f7ebffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.258+0000 7f7ebffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.258+0000 7f7ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.258+0000 7f7ebffff640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:46:27.259 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.258+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4003520 con 0x7f7ec813f710 2024-03-25T19:46:27.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.261+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915724428 0 0) 0x7f7ec8142520 con 0x7f7ec80a98f0 2024-03-25T19:46:27.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.261+0000 7f7ebffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.261+0000 7f7ebffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.261+0000 7f7ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.261+0000 7f7ebffff640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:46:27.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.261+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec80b2a30 con 0x7f7ec80a98f0 2024-03-25T19:46:27.266 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.267+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875035223 0 0) 0x7f7ec80b28a0 con 0x7f7ec81430c0 2024-03-25T19:46:27.266 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.267+0000 7f7ebffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.266 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.267+0000 7f7ebffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.267 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.267+0000 7f7ebffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.267 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.267+0000 7f7ebffff640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:46:27.267 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.267+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8142520 con 0x7f7ec81430c0 2024-03-25T19:46:27.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 829815196 0 0) 0x7f7eb4003520 con 0x7f7ec813f710 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 >> v1:172.21.15.76:6789/0 conn(0x7f7ec81430c0 legacy=0x7f7ec814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 >> v1:172.21.15.50:6789/0 conn(0x7f7ec80a98f0 legacy=0x7f7ec80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 1 monclient: found mon.noname-b 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80b44c0 con 0x7f7ec813f710 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 10 monclient: _finish_auth 0 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:27.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7eb8001690 con 0x7f7ec813f710 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.274+0000 7f7ebffff640 10 monclient: dump: 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:27.278 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:27.279 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.275+0000 7f7ecf8c1640 5 monclient: authenticate success, global_id 4457 2024-03-25T19:46:27.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.275+0000 7f7ecf8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:27.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb80017f0 con 0x7f7ec813f710 2024-03-25T19:46:27.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ebffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:27.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ebffff640 1 -- 172.21.15.50:0/34938774 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7eb8001c00 con 0x7f7ec813f710 2024-03-25T19:46:27.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ecf8c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:27.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ecf8c1640 4 set_mon_vals no callback set 2024-03-25T19:46:27.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ebffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.289 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ebffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.289+0000 7f7ebffff640 10 monclient: dump: 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:27.290 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.290+0000 7f7ecf8c1640 10 monclient: shutdown 2024-03-25T19:46:27.291 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.290+0000 7f7ecf8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.290+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/34938774 >> v1:172.21.15.50:6790/0 conn(0x7f7ec813f710 legacy=0x7f7ec81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.291+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/34938774 shutdown_connections 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.291+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/34938774 >> 172.21.15.50:0/34938774 conn(0x7f7ec80b7ea0 msgr2=0x7f7ec80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.291+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/34938774 shutdown_connections 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.291+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/34938774 wait complete. 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.292+0000 7f7ecf8c1640 10 monclient: build_initial_monmap 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.292+0000 7f7ecf8c1640 10 monclient: monmap: 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:27.292 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:27.293 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:27.293 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:27.293 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.292+0000 7f7ecf8c1640 1 librados: starting msgr at 2024-03-25T19:46:27.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.292+0000 7f7ecf8c1640 1 librados: starting objecter 2024-03-25T19:46:27.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.293+0000 7f7ecf8c1640 1 Processor -- start 2024-03-25T19:46:27.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.293+0000 7f7ecf8c1640 1 -- start start 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.293+0000 7f7ecf8c1640 1 librados: setting wanted keys 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.293+0000 7f7ecf8c1640 1 librados: calling monclient init 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.293+0000 7f7ecf8c1640 10 monclient: init 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec80a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec81430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8137df0 con 0x7f7ec813f710 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec81337a0 con 0x7f7ec81430c0 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8133930 con 0x7f7ec80a98f0 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:27.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.294+0000 7f7ecf8c1640 10 monclient(hunting): authenticate will time out at 5162.557829s 2024-03-25T19:46:27.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.311+0000 7f7ecd65d640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f7ec80a98f0 0x7f7ec81392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:35040/0 (socket says 172.21.15.50:35040) 2024-03-25T19:46:27.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.311+0000 7f7ecd65d640 1 -- 172.21.15.50:0/62956130 learned_addr learned my addr 172.21.15.50:0/62956130 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:27.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.312+0000 7f7ebdffb640 10 client.?.objecter ms_handle_connect 0x7f7ec80a98f0 2024-03-25T19:46:27.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.312+0000 7f7ebdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:27.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.317+0000 7f7ebdffb640 10 client.?.objecter ms_handle_connect 0x7f7ec813f710 2024-03-25T19:46:27.317 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.317+0000 7f7ebdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:27.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.318+0000 7f7ebdffb640 10 client.?.objecter ms_handle_connect 0x7f7ec81430c0 2024-03-25T19:46:27.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.318+0000 7f7ebdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:27.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.324+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917959905 0 0) 0x7f7ec8133930 con 0x7f7ec80a98f0 2024-03-25T19:46:27.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.324+0000 7f7ebdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.324+0000 7f7ebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.324+0000 7f7ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.324+0000 7f7ebdffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:46:27.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.324+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eac004eb0 con 0x7f7ec80a98f0 2024-03-25T19:46:27.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.327+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19819557 0 0) 0x7f7ec81337a0 con 0x7f7ec81430c0 2024-03-25T19:46:27.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.327+0000 7f7ebdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.327+0000 7f7ebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.327+0000 7f7ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.327+0000 7f7ebdffb640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:46:27.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.327+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8133930 con 0x7f7ec81430c0 2024-03-25T19:46:27.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.332+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581045230 0 0) 0x7f7ec8137df0 con 0x7f7ec813f710 2024-03-25T19:46:27.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.332+0000 7f7ebdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:27.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.332+0000 7f7ebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:27.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.332+0000 7f7ebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:27.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.332+0000 7f7ebdffb640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:46:27.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.332+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec81337a0 con 0x7f7ec813f710 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.352+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3458790191 0 0) 0x7f7eac004eb0 con 0x7f7ec80a98f0 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.352+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec8137df0 con 0x7f7ec80a98f0 2024-03-25T19:46:27.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.352+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7ec4001a30 con 0x7f7ec80a98f0 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.352+0000 7f7ebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.352+0000 7f7ebdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.352+0000 7f7ebdffb640 10 monclient(hunting): dump: 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.352+0000 7f7ebdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:27.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.353+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077522961 0 0) 0x7f7ec8133930 con 0x7f7ec81430c0 2024-03-25T19:46:27.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.353+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7eac004eb0 con 0x7f7ec81430c0 2024-03-25T19:46:27.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.353+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7ec0001b50 con 0x7f7ec81430c0 2024-03-25T19:46:27.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.353+0000 7f7ebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.353+0000 7f7ebdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:27.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.353+0000 7f7ebdffb640 10 monclient(hunting): dump: 2024-03-25T19:46:27.353 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:27.354 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.356+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3045438287 0 0) 0x7f7ec81337a0 con 0x7f7ec813f710 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.356+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec8133930 con 0x7f7ec813f710 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.356+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7eb8001b00 con 0x7f7ec813f710 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.356+0000 7f7ebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.356+0000 7f7ebdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.356+0000 7f7ebdffb640 10 monclient(hunting): dump: 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:27.356 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:27.357 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:27.357 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:27.357 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2062721452 0 0) 0x7f7eac004eb0 con 0x7f7ec81430c0 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 >> v1:172.21.15.50:6790/0 conn(0x7f7ec80a98f0 legacy=0x7f7ec81392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 >> v1:172.21.15.50:6789/0 conn(0x7f7ec813f710 legacy=0x7f7ec81376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 1 monclient: found mon.b 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec81367d0 con 0x7f7ec81430c0 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ebdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 10 monclient: _renew_subs 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:27.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec81369e0 con 0x7f7ec81430c0 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 10 monclient: _renew_subs 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec813b640 con 0x7f7ec81430c0 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.374+0000 7f7ecf8c1640 1 librados: init done 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.375+0000 7f7ecf8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.375+0000 7f7ecf8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:27.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.375+0000 7f7ecf8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:27.377 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.375+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8074cc0 con 0x7f7ec81430c0 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.650+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ec0001fe0 con 0x7f7ec81430c0 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.650+0000 7f7ebdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.650+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7ec0003b10 con 0x7f7ec81430c0 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.650+0000 7f7ebdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.650+0000 7f7ebdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.650+0000 7f7ebdffb640 10 monclient: dump: 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.650 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:27.651 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:27.651 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:27.651 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:27.651 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:27.651 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:27.651 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:27.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.650+0000 7f7ebf7fe640 4 set_mon_vals no callback set 2024-03-25T19:46:27.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.651+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f7ec0019ae0 con 0x7f7ec81430c0 2024-03-25T19:46:27.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.651+0000 7f7ecd65d640 10 monclient: get_auth_request con 0x7f7eac06e130 auth_method 0 2024-03-25T19:46:27.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.652+0000 7f7ebdffb640 10 client.4465.objecter ms_handle_connect 0x7f7eac06e130 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f7ec00537f0 con 0x7f7ec81430c0 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 10 client.4465.objecter ms_dispatch 0x7f7ec8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ec004f130 con 0x7f7ec81430c0 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:27.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.656+0000 7f7ebdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:27.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.915+0000 7f7ecf8c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:27.915 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.915+0000 7f7ecf8c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:27.916 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.915+0000 7f7ecf8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:27.916 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:27.916+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ec80627b0 con 0x7f7ec81430c0 2024-03-25T19:46:28.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.002+0000 7f7ebdffb640 1 -- 172.21.15.50:0/62956130 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f7ec0051230 con 0x7f7ec81430c0 2024-03-25T19:46:28.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.002+0000 7f7ebdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:28.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.002+0000 7f7ebdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:28.002 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:28.002 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:28.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.004+0000 7f7ecf8c1640 10 librados: watch_flush enter 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.004+0000 7f7ecf8c1640 10 librados: watch_flush exit 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.004+0000 7f7ecf8c1640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.004+0000 7f7ecf8c1640 10 client.4465.objecter successfully canceled tick 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 >> v1:172.21.15.76:6800/2572556811 conn(0x7f7eac06e130 legacy=0x7f7eac0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 10 monclient: shutdown 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 >> v1:172.21.15.76:6789/0 conn(0x7f7ec81430c0 legacy=0x7f7ec8133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 shutdown_connections 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 >> 172.21.15.50:0/62956130 conn(0x7f7ec80b7ea0 msgr2=0x7f7ec8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 shutdown_connections 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 1 -- 172.21.15.50:0/62956130 wait complete. 2024-03-25T19:46:28.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:28.005+0000 7f7ecf8c1640 1 librados: shutdown 2024-03-25T19:46:28.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:34.823 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.936+0000 7fbf5a332640 10 monclient: get_monmap_and_config 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.936+0000 7fbf5a332640 10 monclient: build_initial_monmap 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.936+0000 7fbf5a332640 10 monclient: monmap: 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:34.937127+0000 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:34.937127+0000 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:34.936 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:34.937 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:34.937 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:34.937 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 1 Processor -- start 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 1 -- start start 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 10 monclient: init 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 10 monclient(hunting): picked mon.noname-b con 0x7fbf540a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 10 monclient(hunting): picked mon.noname-c con 0x7fbf541430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.937+0000 7fbf5a332640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf54142520 con 0x7fbf5413f710 2024-03-25T19:46:34.938 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.938+0000 7fbf5a332640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540b28a0 con 0x7fbf541430c0 2024-03-25T19:46:34.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.938+0000 7fbf5a332640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540b2a30 con 0x7fbf540a98f0 2024-03-25T19:46:34.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.938+0000 7fbf5a332640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:34.939 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.938+0000 7fbf5a332640 10 monclient(hunting): authenticate will time out at 5170.201730s 2024-03-25T19:46:34.943 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.944+0000 7fbf588cf640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbf541430c0 0x7fbf5414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:37986/0 (socket says 172.21.15.50:37986) 2024-03-25T19:46:34.943 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.944+0000 7fbf588cf640 1 -- 172.21.15.50:0/287392856 learned_addr learned my addr 172.21.15.50:0/287392856 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.955+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569945419 0 0) 0x7fbf540b28a0 con 0x7fbf541430c0 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.955+0000 7fbf52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.955+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.955+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.955+0000 7fbf52ffd640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:46:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.955+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf3c003520 con 0x7fbf541430c0 2024-03-25T19:46:34.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.974+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913776321 0 0) 0x7fbf540b2a30 con 0x7fbf540a98f0 2024-03-25T19:46:34.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.974+0000 7fbf52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.974+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.974+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.974+0000 7fbf52ffd640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:46:34.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.974+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf540b28a0 con 0x7fbf540a98f0 2024-03-25T19:46:34.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.981+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512993678 0 0) 0x7fbf54142520 con 0x7fbf5413f710 2024-03-25T19:46:34.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.981+0000 7fbf52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:34.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.981+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:34.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.981+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:34.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.981+0000 7fbf52ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:46:34.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:34.981+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf540b2a30 con 0x7fbf5413f710 2024-03-25T19:46:35.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.048+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116387606 0 0) 0x7fbf3c003520 con 0x7fbf541430c0 2024-03-25T19:46:35.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 >> v1:172.21.15.50:6790/0 conn(0x7fbf540a98f0 legacy=0x7fbf540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 >> v1:172.21.15.50:6789/0 conn(0x7fbf5413f710 legacy=0x7fbf541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 1 monclient: found mon.noname-c 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540b44c0 con 0x7fbf541430c0 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbf4c001a10 con 0x7fbf541430c0 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf52ffd640 10 monclient: dump: 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:35.050 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf5a332640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:46:35.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.049+0000 7fbf5a332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:35.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf4c001b70 con 0x7fbf541430c0 2024-03-25T19:46:35.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:35.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf52ffd640 1 -- 172.21.15.50:0/287392856 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbf4c001f80 con 0x7fbf541430c0 2024-03-25T19:46:35.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf52ffd640 10 monclient: dump: 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:35.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 4 set_mon_vals no callback set 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 10 monclient: shutdown 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 1 -- 172.21.15.50:0/287392856 >> v1:172.21.15.76:6789/0 conn(0x7fbf541430c0 legacy=0x7fbf5414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 1 -- 172.21.15.50:0/287392856 shutdown_connections 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 1 -- 172.21.15.50:0/287392856 >> 172.21.15.50:0/287392856 conn(0x7fbf540b7ea0 msgr2=0x7fbf540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 1 -- 172.21.15.50:0/287392856 shutdown_connections 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.064+0000 7fbf5a332640 1 -- 172.21.15.50:0/287392856 wait complete. 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.065+0000 7fbf5a332640 10 monclient: build_initial_monmap 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.065+0000 7fbf5a332640 10 monclient: monmap: 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:35.066 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.065+0000 7fbf5a332640 1 librados: starting msgr at 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.065+0000 7fbf5a332640 1 librados: starting objecter 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.066+0000 7fbf5a332640 1 Processor -- start 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.066+0000 7fbf5a332640 1 -- start start 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.066+0000 7fbf5a332640 1 librados: setting wanted keys 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.066+0000 7fbf5a332640 1 librados: calling monclient init 2024-03-25T19:46:35.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.066+0000 7fbf5a332640 10 monclient: init 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 10 monclient(hunting): picked mon.noname-c con 0x7fbf540a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 10 monclient(hunting): picked mon.noname-a con 0x7fbf541430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf54135230 con 0x7fbf541430c0 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf54135da0 con 0x7fbf5413f710 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf54135f30 con 0x7fbf540a98f0 2024-03-25T19:46:35.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:35.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf5a332640 10 monclient(hunting): authenticate will time out at 5170.330963s 2024-03-25T19:46:35.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.067+0000 7fbf537fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbf5413f710 0x7fbf541dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:38018/0 (socket says 172.21.15.50:38018) 2024-03-25T19:46:35.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.068+0000 7fbf537fe640 1 -- 172.21.15.50:0/4032069966 learned_addr learned my addr 172.21.15.50:0/4032069966 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:35.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.068+0000 7fbf50ff9640 10 client.?.objecter ms_handle_connect 0x7fbf5413f710 2024-03-25T19:46:35.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.068+0000 7fbf50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.069+0000 7fbf50ff9640 10 client.?.objecter ms_handle_connect 0x7fbf540a98f0 2024-03-25T19:46:35.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.069+0000 7fbf50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.087+0000 7fbf50ff9640 10 client.?.objecter ms_handle_connect 0x7fbf541430c0 2024-03-25T19:46:35.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.087+0000 7fbf50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:35.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.141+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197788545 0 0) 0x7fbf54135230 con 0x7fbf541430c0 2024-03-25T19:46:35.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.141+0000 7fbf50ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.141+0000 7fbf50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.141+0000 7fbf50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.141+0000 7fbf50ff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:46:35.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.141+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf40004eb0 con 0x7fbf541430c0 2024-03-25T19:46:35.147 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.148+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430823244 0 0) 0x7fbf54135da0 con 0x7fbf5413f710 2024-03-25T19:46:35.147 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.148+0000 7fbf50ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:35.147 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.148+0000 7fbf50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:35.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.148+0000 7fbf50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:35.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.148+0000 7fbf50ff9640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:46:35.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.148+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54135230 con 0x7fbf5413f710 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.155+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2236291365 0 0) 0x7fbf40004eb0 con 0x7fbf541430c0 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.155+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf54135da0 con 0x7fbf541430c0 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.156+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbf4c001e80 con 0x7fbf541430c0 2024-03-25T19:46:35.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.156+0000 7fbf50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.156+0000 7fbf50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.156+0000 7fbf50ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:35.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.156+0000 7fbf50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.165+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2299774619 0 0) 0x7fbf54135da0 con 0x7fbf541430c0 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 >> v1:172.21.15.50:6790/0 conn(0x7fbf540a98f0 legacy=0x7fbf541d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 >> v1:172.21.15.76:6789/0 conn(0x7fbf5413f710 legacy=0x7fbf541dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 1 monclient: found mon.a 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf541360c0 con 0x7fbf541430c0 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:35.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 5 monclient: authenticate success, global_id 4491 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 10 monclient: _renew_subs 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf541d5000 con 0x7fbf541430c0 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 10 monclient: _renew_subs 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf541d5350 con 0x7fbf541430c0 2024-03-25T19:46:35.167 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.166+0000 7fbf5a332640 1 librados: init done 2024-03-25T19:46:35.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.167+0000 7fbf5a332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.167+0000 7fbf5a332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.167+0000 7fbf5a332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:35.168 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.167+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf54135090 con 0x7fbf541430c0 2024-03-25T19:46:35.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.488+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf4c002020 con 0x7fbf541430c0 2024-03-25T19:46:35.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.488+0000 7fbf50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.488+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbf4c004e10 con 0x7fbf541430c0 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.488+0000 7fbf50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.488+0000 7fbf50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.488+0000 7fbf50ff9640 10 monclient: dump: 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:35.489 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:35.490 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:35.490 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:35.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.488+0000 7fbf527fc640 4 set_mon_vals no callback set 2024-03-25T19:46:35.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.489+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fbf4c01ade0 con 0x7fbf541430c0 2024-03-25T19:46:35.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.490+0000 7fbf53fff640 10 monclient: get_auth_request con 0x7fbf4006dcc0 auth_method 0 2024-03-25T19:46:35.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.490+0000 7fbf50ff9640 10 client.4491.objecter ms_handle_connect 0x7fbf4006dcc0 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fbf4c054930 con 0x7fbf541430c0 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 10 client.4491.objecter ms_dispatch 0x7fbf54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf4c050270 con 0x7fbf541430c0 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:35.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.495+0000 7fbf50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:35.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.749+0000 7fbf5a332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:35.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.749+0000 7fbf5a332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:35.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.749+0000 7fbf5a332640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:35.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.749+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf541d44e0 con 0x7fbf541430c0 2024-03-25T19:46:35.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.787+0000 7fbf50ff9640 1 -- 172.21.15.50:0/4032069966 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fbf4c052370 con 0x7fbf541430c0 2024-03-25T19:46:35.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.787+0000 7fbf50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:35.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.787+0000 7fbf50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:35.788 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:35.788 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 10 librados: watch_flush enter 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 10 librados: watch_flush exit 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 10 client.4491.objecter successfully canceled tick 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 >> v1:172.21.15.76:6800/2572556811 conn(0x7fbf4006dcc0 legacy=0x7fbf40070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 10 monclient: shutdown 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 >> v1:172.21.15.50:6789/0 conn(0x7fbf541430c0 legacy=0x7fbf541db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 shutdown_connections 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 >> 172.21.15.50:0/4032069966 conn(0x7fbf540b7ea0 msgr2=0x7fbf5414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 shutdown_connections 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 1 -- 172.21.15.50:0/4032069966 wait complete. 2024-03-25T19:46:35.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:35.790+0000 7fbf5a332640 1 librados: shutdown 2024-03-25T19:46:35.802 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:40.062 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:00.567 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.072 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:00.578 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.115 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:00.621 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.120 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:00.591 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.124 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:00.595 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.142 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:00.647 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.149 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:00.655 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.150 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:00.620 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.176 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:00.681 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:42.609 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.723+0000 7ff8b5863640 10 monclient: get_monmap_and_config 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.723+0000 7ff8b5863640 10 monclient: build_initial_monmap 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.723+0000 7ff8b5863640 10 monclient: monmap: 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:42.724207+0000 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:42.724207+0000 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:42.723 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:42.724 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:42.724 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:42.724 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:42.724 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 1 Processor -- start 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 1 -- start start 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 10 monclient: init 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 10 monclient(hunting): picked mon.noname-c con 0x7ff8b00a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 10 monclient(hunting): picked mon.noname-b con 0x7ff8b013f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 10 monclient(hunting): picked mon.noname-a con 0x7ff8b01430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.724+0000 7ff8b5863640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b0142520 con 0x7ff8b01430c0 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.725+0000 7ff8b5863640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b00b28a0 con 0x7ff8b00a98f0 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.725+0000 7ff8b5863640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b00b2a30 con 0x7ff8b013f710 2024-03-25T19:46:42.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.725+0000 7ff8b5863640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:42.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.725+0000 7ff8b5863640 10 monclient(hunting): authenticate will time out at 5177.988742s 2024-03-25T19:46:42.730 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.731+0000 7ff8a7fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7ff8b013f710 0x7ff8b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:56428/0 (socket says 172.21.15.50:56428) 2024-03-25T19:46:42.730 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.731+0000 7ff8a7fff640 1 -- 172.21.15.50:0/105101947 learned_addr learned my addr 172.21.15.50:0/105101947 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:42.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.746+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615223380 0 0) 0x7ff8b00b28a0 con 0x7ff8b00a98f0 2024-03-25T19:46:42.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.746+0000 7ff8ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.746+0000 7ff8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.746+0000 7ff8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.746+0000 7ff8ae7fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:46:42.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.746+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff89c003520 con 0x7ff8b00a98f0 2024-03-25T19:46:42.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.747+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032712255 0 0) 0x7ff8b00b2a30 con 0x7ff8b013f710 2024-03-25T19:46:42.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.747+0000 7ff8ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.747+0000 7ff8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.747+0000 7ff8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.747+0000 7ff8ae7fc640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:46:42.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.747+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b00b28a0 con 0x7ff8b013f710 2024-03-25T19:46:42.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005115725 0 0) 0x7ff8b00b28a0 con 0x7ff8b013f710 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 >> v1:172.21.15.76:6789/0 conn(0x7ff8b00a98f0 legacy=0x7ff8b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 >> v1:172.21.15.50:6789/0 conn(0x7ff8b01430c0 legacy=0x7ff8b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 1 monclient: found mon.noname-b 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b00b44c0 con 0x7ff8b013f710 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff898001670 con 0x7ff8b013f710 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8ae7fc640 10 monclient: dump: 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8b5863640 5 monclient: authenticate success, global_id 4469 2024-03-25T19:46:42.763 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.762+0000 7ff8b5863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:42.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8980017d0 con 0x7ff8b013f710 2024-03-25T19:46:42.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:42.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8ae7fc640 1 -- 172.21.15.50:0/105101947 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff898001c20 con 0x7ff8b013f710 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8ae7fc640 10 monclient: dump: 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:42.770 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:42.771 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:42.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:42.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 4 set_mon_vals no callback set 2024-03-25T19:46:42.772 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 10 monclient: shutdown 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 1 -- 172.21.15.50:0/105101947 >> v1:172.21.15.50:6790/0 conn(0x7ff8b013f710 legacy=0x7ff8b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 1 -- 172.21.15.50:0/105101947 shutdown_connections 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 1 -- 172.21.15.50:0/105101947 >> 172.21.15.50:0/105101947 conn(0x7ff8b00b7ea0 msgr2=0x7ff8b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 1 -- 172.21.15.50:0/105101947 shutdown_connections 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.770+0000 7ff8b5863640 1 -- 172.21.15.50:0/105101947 wait complete. 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.771+0000 7ff8b5863640 10 monclient: build_initial_monmap 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.771+0000 7ff8b5863640 10 monclient: monmap: 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:42.773 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.771+0000 7ff8b5863640 1 librados: starting msgr at 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.771+0000 7ff8b5863640 1 librados: starting objecter 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 1 Processor -- start 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 1 -- start start 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 1 librados: setting wanted keys 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 1 librados: calling monclient init 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient: init 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:42.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient(hunting): picked mon.noname-c con 0x7ff8b00a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient(hunting): picked mon.noname-a con 0x7ff8b013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient(hunting): picked mon.noname-b con 0x7ff8b01430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b013b720 con 0x7ff8b013f710 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b0139d20 con 0x7ff8b01430c0 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b00b7b70 con 0x7ff8b00a98f0 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:42.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.772+0000 7ff8b5863640 10 monclient(hunting): authenticate will time out at 5178.036465s 2024-03-25T19:46:42.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.777+0000 7ff8af7fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff8b01430c0 0x7ff8b01e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:38606/0 (socket says 172.21.15.50:38606) 2024-03-25T19:46:42.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.777+0000 7ff8af7fe640 1 -- 172.21.15.50:0/3706153927 learned_addr learned my addr 172.21.15.50:0/3706153927 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:42.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.777+0000 7ff8acff9640 10 client.?.objecter ms_handle_connect 0x7ff8b01430c0 2024-03-25T19:46:42.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.777+0000 7ff8acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:42.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.785+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442229940 0 0) 0x7ff8b0139d20 con 0x7ff8b01430c0 2024-03-25T19:46:42.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.785+0000 7ff8acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.785+0000 7ff8acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.785+0000 7ff8acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.785+0000 7ff8acff9640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:46:42.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.785+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890004eb0 con 0x7ff8b01430c0 2024-03-25T19:46:42.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.788+0000 7ff8acff9640 10 client.?.objecter ms_handle_connect 0x7ff8b00a98f0 2024-03-25T19:46:42.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.788+0000 7ff8acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:42.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.792+0000 7ff8acff9640 10 client.?.objecter ms_handle_connect 0x7ff8b013f710 2024-03-25T19:46:42.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.792+0000 7ff8acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938651601 0 0) 0x7ff8b013b720 con 0x7ff8b013f710 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b0139d20 con 0x7ff8b013f710 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1374167271 0 0) 0x7ff890004eb0 con 0x7ff8b01430c0 2024-03-25T19:46:42.804 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8b013b720 con 0x7ff8b01430c0 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff8a8001c90 con 0x7ff8b01430c0 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): dump: 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:42.805 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:42.806 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:42.806 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.804+0000 7ff8acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:42.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.813+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553855210 0 0) 0x7ff8b00b7b70 con 0x7ff8b00a98f0 2024-03-25T19:46:42.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.813+0000 7ff8acff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.813+0000 7ff8acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.813+0000 7ff8acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.813+0000 7ff8acff9640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:46:42.813 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.813+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890004eb0 con 0x7ff8b00a98f0 2024-03-25T19:46:42.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.821+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19120212 0 0) 0x7ff8b0139d20 con 0x7ff8b013f710 2024-03-25T19:46:42.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.821+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8b00b7b70 con 0x7ff8b013f710 2024-03-25T19:46:42.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.821+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff898001b20 con 0x7ff8b013f710 2024-03-25T19:46:42.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.821+0000 7ff8acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:42.821 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.821+0000 7ff8acff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.821+0000 7ff8acff9640 10 monclient(hunting): dump: 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:42.822 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:42.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 22959912 0 0) 0x7ff8b00b7b70 con 0x7ff8b013f710 2024-03-25T19:46:42.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 >> v1:172.21.15.50:6790/0 conn(0x7ff8b00a98f0 legacy=0x7ff8b013bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 >> v1:172.21.15.76:6789/0 conn(0x7ff8b01430c0 legacy=0x7ff8b01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 1 monclient: found mon.a 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b013c250 con 0x7ff8b013f710 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 10 monclient: _renew_subs 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b00b71f0 con 0x7ff8b013f710 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 10 monclient: _renew_subs 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b012f910 con 0x7ff8b013f710 2024-03-25T19:46:42.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.835+0000 7ff8b5863640 1 librados: init done 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.836+0000 7ff8b5863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.836+0000 7ff8b5863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.836+0000 7ff8b5863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:42.837 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:42.836+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b0139eb0 con 0x7ff8b013f710 2024-03-25T19:46:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.149+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff898004a90 con 0x7ff8b013f710 2024-03-25T19:46:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.149+0000 7ff8acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:43.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.149+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff898004da0 con 0x7ff8b013f710 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.149+0000 7ff8acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.149+0000 7ff8acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.149+0000 7ff8acff9640 10 monclient: dump: 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:43.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.149+0000 7ff8adffb640 4 set_mon_vals no callback set 2024-03-25T19:46:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.150+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7ff89801ad70 con 0x7ff8b013f710 2024-03-25T19:46:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.151+0000 7ff8aeffd640 10 monclient: get_auth_request con 0x7ff89006df70 auth_method 0 2024-03-25T19:46:43.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.151+0000 7ff8acff9640 10 client.4497.objecter ms_handle_connect 0x7ff89006df70 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7ff898054830 con 0x7ff8b013f710 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 10 client.4497.objecter ms_dispatch 0x7ff8b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff898050340 con 0x7ff8b013f710 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:43.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.158+0000 7ff8acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:43.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.405+0000 7ff8b5863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:43.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.405+0000 7ff8b5863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:43.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.405+0000 7ff8b5863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:46:43.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.405+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8b013c3e0 con 0x7ff8b013f710 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.430+0000 7ff8acff9640 1 -- 172.21.15.50:0/3706153927 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7ff898052270 con 0x7ff8b013f710 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.430+0000 7ff8acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.430+0000 7ff8acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:43.430 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:43.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.432+0000 7ff8b5863640 10 librados: watch_flush enter 2024-03-25T19:46:43.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 10 librados: watch_flush exit 2024-03-25T19:46:43.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T19:46:43.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 10 client.4497.objecter successfully canceled tick 2024-03-25T19:46:43.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 >> v1:172.21.15.76:6800/2572556811 conn(0x7ff89006df70 legacy=0x7ff890070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 10 monclient: shutdown 2024-03-25T19:46:43.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:43.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 >> v1:172.21.15.50:6789/0 conn(0x7ff8b013f710 legacy=0x7ff8b01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:43.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 shutdown_connections 2024-03-25T19:46:43.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 >> 172.21.15.50:0/3706153927 conn(0x7ff8b00b7ea0 msgr2=0x7ff8b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:43.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 shutdown_connections 2024-03-25T19:46:43.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 1 -- 172.21.15.50:0/3706153927 wait complete. 2024-03-25T19:46:43.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:43.433+0000 7ff8b5863640 1 librados: shutdown 2024-03-25T19:46:43.446 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:50.253 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:50.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.366+0000 7fb2eee88640 10 monclient: get_monmap_and_config 2024-03-25T19:46:50.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.366+0000 7fb2eee88640 10 monclient: build_initial_monmap 2024-03-25T19:46:50.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.366+0000 7fb2eee88640 10 monclient: monmap: 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:50.367668+0000 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:50.367668+0000 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:50.367 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.367+0000 7fb2eee88640 1 Processor -- start 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.367+0000 7fb2eee88640 1 -- start start 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.367+0000 7fb2eee88640 10 monclient: init 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 10 monclient(hunting): picked mon.noname-b con 0x7fb2e80b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 10 monclient(hunting): picked mon.noname-a con 0x7fb2e80b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 10 monclient(hunting): picked mon.noname-c con 0x7fb2e814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80af940 con 0x7fb2e80b1730 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80b0a70 con 0x7fb2e814be90 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80b0d50 con 0x7fb2e80b4d60 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:50.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.368+0000 7fb2eee88640 10 monclient(hunting): authenticate will time out at 5185.632523s 2024-03-25T19:46:50.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.381+0000 7fb2ed425640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb2e814be90 0x7fb2e814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:36090/0 (socket says 172.21.15.50:36090) 2024-03-25T19:46:50.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.381+0000 7fb2ed425640 1 -- 172.21.15.50:0/3563241301 learned_addr learned my addr 172.21.15.50:0/3563241301 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:50.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.390+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175026136 0 0) 0x7fb2e80b0d50 con 0x7fb2e80b4d60 2024-03-25T19:46:50.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.390+0000 7fb2e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.390+0000 7fb2e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.390+0000 7fb2e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.390+0000 7fb2e77fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:46:50.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.390+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d8003520 con 0x7fb2e80b4d60 2024-03-25T19:46:50.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.395+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610492638 0 0) 0x7fb2e80b0a70 con 0x7fb2e814be90 2024-03-25T19:46:50.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.395+0000 7fb2e77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.395+0000 7fb2e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.395+0000 7fb2e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.395+0000 7fb2e77fe640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:46:50.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.395+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e80b0d50 con 0x7fb2e814be90 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103168464 0 0) 0x7fb2d8003520 con 0x7fb2e80b4d60 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 >> v1:172.21.15.76:6789/0 conn(0x7fb2e814be90 legacy=0x7fb2e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 >> v1:172.21.15.50:6789/0 conn(0x7fb2e80b1730 legacy=0x7fb2e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 1 monclient: found mon.noname-b 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:46:50.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80b1030 con 0x7fb2e80b4d60 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb2dc001690 con 0x7fb2e80b4d60 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2eee88640 5 monclient: authenticate success, global_id 4475 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2eee88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2e77fe640 10 monclient: dump: 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.403 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.404 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.404 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:50.404 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:50.404 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:50.404 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:50.404 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:50.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.402+0000 7fb2eee88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:50.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2dc0017f0 con 0x7fb2e80b4d60 2024-03-25T19:46:50.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:50.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2e77fe640 1 -- 172.21.15.50:0/3563241301 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb2dc001c00 con 0x7fb2e80b4d60 2024-03-25T19:46:50.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:50.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2e77fe640 10 monclient: dump: 2024-03-25T19:46:50.414 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2eee88640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:50.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.414+0000 7fb2eee88640 4 set_mon_vals no callback set 2024-03-25T19:46:50.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.415+0000 7fb2eee88640 10 monclient: shutdown 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.415+0000 7fb2eee88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.415+0000 7fb2eee88640 1 -- 172.21.15.50:0/3563241301 >> v1:172.21.15.50:6790/0 conn(0x7fb2e80b4d60 legacy=0x7fb2e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.415+0000 7fb2eee88640 1 -- 172.21.15.50:0/3563241301 shutdown_connections 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.415+0000 7fb2eee88640 1 -- 172.21.15.50:0/3563241301 >> 172.21.15.50:0/3563241301 conn(0x7fb2e80617a0 msgr2=0x7fb2e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.415+0000 7fb2eee88640 1 -- 172.21.15.50:0/3563241301 shutdown_connections 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.415+0000 7fb2eee88640 1 -- 172.21.15.50:0/3563241301 wait complete. 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 10 monclient: build_initial_monmap 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 10 monclient: monmap: 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:50.418 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 1 librados: starting msgr at 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 1 librados: starting objecter 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 1 Processor -- start 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 1 -- start start 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 1 librados: setting wanted keys 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 1 librados: calling monclient init 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.416+0000 7fb2eee88640 10 monclient: init 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 10 monclient(hunting): picked mon.noname-b con 0x7fb2e80b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:50.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 10 monclient(hunting): picked mon.noname-c con 0x7fb2e80b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:50.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 10 monclient(hunting): picked mon.noname-a con 0x7fb2e814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:50.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e8141bf0 con 0x7fb2e814be90 2024-03-25T19:46:50.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e8141ed0 con 0x7fb2e80b1730 2024-03-25T19:46:50.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e81421b0 con 0x7fb2e80b4d60 2024-03-25T19:46:50.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:50.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.417+0000 7fb2eee88640 10 monclient(hunting): authenticate will time out at 5185.681344s 2024-03-25T19:46:50.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.423+0000 7fb2ecc24640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb2e80b1730 0x7fb2e81425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:36132/0 (socket says 172.21.15.50:36132) 2024-03-25T19:46:50.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.423+0000 7fb2ecc24640 1 -- 172.21.15.50:0/2220365828 learned_addr learned my addr 172.21.15.50:0/2220365828 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:50.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.425+0000 7fb2e57fa640 10 client.?.objecter ms_handle_connect 0x7fb2e80b1730 2024-03-25T19:46:50.424 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.425+0000 7fb2e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:50.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.437+0000 7fb2e57fa640 10 client.?.objecter ms_handle_connect 0x7fb2e80b4d60 2024-03-25T19:46:50.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.437+0000 7fb2e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:50.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.438+0000 7fb2e57fa640 10 client.?.objecter ms_handle_connect 0x7fb2e814be90 2024-03-25T19:46:50.438 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.438+0000 7fb2e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:50.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.442+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185732681 0 0) 0x7fb2e8141ed0 con 0x7fb2e80b1730 2024-03-25T19:46:50.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.442+0000 7fb2e57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.442+0000 7fb2e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.442+0000 7fb2e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.442+0000 7fb2e57fa640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:46:50.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.442+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2cc004eb0 con 0x7fb2e80b1730 2024-03-25T19:46:50.448 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.449+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514959453 0 0) 0x7fb2e8141bf0 con 0x7fb2e814be90 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.449+0000 7fb2e57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.449+0000 7fb2e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.449+0000 7fb2e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.449+0000 7fb2e57fa640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:46:50.449 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.449+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e8141ed0 con 0x7fb2e814be90 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.451+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956079032 0 0) 0x7fb2cc004eb0 con 0x7fb2e80b1730 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.451+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2e8141bf0 con 0x7fb2e80b1730 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.451+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb2dc001bc0 con 0x7fb2e80b1730 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.451+0000 7fb2e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.451+0000 7fb2e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.451+0000 7fb2e57fa640 10 monclient(hunting): dump: 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:50.451 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:50.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.451+0000 7fb2e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:50.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.452+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017994598 0 0) 0x7fb2e81421b0 con 0x7fb2e80b4d60 2024-03-25T19:46:50.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.452+0000 7fb2e57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:50.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.452+0000 7fb2e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:50.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.452+0000 7fb2e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:50.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.452+0000 7fb2e57fa640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:46:50.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.452+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2cc004eb0 con 0x7fb2e80b4d60 2024-03-25T19:46:50.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 445925568 0 0) 0x7fb2e8141bf0 con 0x7fb2e80b1730 2024-03-25T19:46:50.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 >> v1:172.21.15.50:6790/0 conn(0x7fb2e80b4d60 legacy=0x7fb2e8133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 >> v1:172.21.15.50:6789/0 conn(0x7fb2e814be90 legacy=0x7fb2e81341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 1 monclient: found mon.b 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8140f00 con 0x7fb2e80b1730 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 10 monclient: _renew_subs 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e81406f0 con 0x7fb2e80b1730 2024-03-25T19:46:50.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:50.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 10 monclient: _renew_subs 2024-03-25T19:46:50.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:50.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e8140380 con 0x7fb2e80b1730 2024-03-25T19:46:50.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.461+0000 7fb2eee88640 1 librados: init done 2024-03-25T19:46:50.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.463+0000 7fb2eee88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:50.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.463+0000 7fb2eee88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:50.463 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.463+0000 7fb2eee88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:50.464 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.463+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e81401f0 con 0x7fb2e80b1730 2024-03-25T19:46:50.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.743+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2dc0011f0 con 0x7fb2e80b1730 2024-03-25T19:46:50.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.743+0000 7fb2e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.743+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb2dc003b00 con 0x7fb2e80b1730 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.743+0000 7fb2e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.743+0000 7fb2e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.743+0000 7fb2e57fa640 10 monclient: dump: 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:50.744 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:50.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.743+0000 7fb2e6ffd640 4 set_mon_vals no callback set 2024-03-25T19:46:50.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.744+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fb2dc003d20 con 0x7fb2e80b1730 2024-03-25T19:46:50.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.744+0000 7fb2e7fff640 10 monclient: get_auth_request con 0x7fb2cc06de00 auth_method 0 2024-03-25T19:46:50.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.745+0000 7fb2e57fa640 10 client.4483.objecter ms_handle_connect 0x7fb2cc06de00 2024-03-25T19:46:50.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.749+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fb2dc054880 con 0x7fb2e80b1730 2024-03-25T19:46:50.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.749+0000 7fb2e57fa640 10 client.4483.objecter ms_dispatch 0x7fb2e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:50.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.749+0000 7fb2e57fa640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:50.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.749+0000 7fb2e57fa640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:50.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.750+0000 7fb2e57fa640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T19:46:50.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.750+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2dc050390 con 0x7fb2e80b1730 2024-03-25T19:46:50.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.750+0000 7fb2e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:50.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:50.750+0000 7fb2e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:51.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.004+0000 7fb2eee88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.004+0000 7fb2eee88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.004+0000 7fb2eee88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:51.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.004+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2e80627b0 con 0x7fb2e80b1730 2024-03-25T19:46:51.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.155+0000 7fb2e57fa640 1 -- 172.21.15.50:0/2220365828 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fb2dc0522c0 con 0x7fb2e80b1730 2024-03-25T19:46:51.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.155+0000 7fb2e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:51.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.155+0000 7fb2e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:51.155 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:51.156 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.157+0000 7fb2eee88640 10 librados: watch_flush enter 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.157+0000 7fb2eee88640 10 librados: watch_flush exit 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 10 client.4483.objecter successfully canceled tick 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 >> v1:172.21.15.76:6800/2572556811 conn(0x7fb2cc06de00 legacy=0x7fb2cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 10 monclient: shutdown 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 >> v1:172.21.15.76:6789/0 conn(0x7fb2e80b1730 legacy=0x7fb2e81425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:51.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 shutdown_connections 2024-03-25T19:46:51.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 >> 172.21.15.50:0/2220365828 conn(0x7fb2e80617a0 msgr2=0x7fb2e813c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:51.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 shutdown_connections 2024-03-25T19:46:51.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 1 -- 172.21.15.50:0/2220365828 wait complete. 2024-03-25T19:46:51.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:51.158+0000 7fb2eee88640 1 librados: shutdown 2024-03-25T19:46:51.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:57.975 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:58.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.093+0000 7fcb740a8640 10 monclient: get_monmap_and_config 2024-03-25T19:46:58.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.093+0000 7fcb740a8640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.093+0000 7fcb740a8640 10 monclient: monmap: 2024-03-25T19:46:58.094 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:58.094 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:46:58.095592+0000 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:46:58.095592+0000 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:46:58.095 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:58.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.094+0000 7fcb740a8640 1 Processor -- start 2024-03-25T19:46:58.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.094+0000 7fcb740a8640 1 -- start start 2024-03-25T19:46:58.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.094+0000 7fcb740a8640 10 monclient: init 2024-03-25T19:46:58.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6c0b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c0b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c14be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0af940 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0b0a70 con 0x7fcb6c0b1730 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0b0d50 con 0x7fcb6c14be90 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.095+0000 7fcb740a8640 10 monclient(hunting): authenticate will time out at 5193.360036s 2024-03-25T19:46:58.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.103+0000 7fcb71e44640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcb6c0b4d60 0x7fcb6c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36178/0 (socket says 172.21.15.50:36178) 2024-03-25T19:46:58.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.103+0000 7fcb71e44640 1 -- 172.21.15.50:0/1913123599 learned_addr learned my addr 172.21.15.50:0/1913123599 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.114+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991087925 0 0) 0x7fcb6c0af940 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.114+0000 7fcb70e42640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.114+0000 7fcb70e42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.114+0000 7fcb70e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.114+0000 7fcb70e42640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:46:58.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.114+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb58003520 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.117+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112841021 0 0) 0x7fcb6c0b0d50 con 0x7fcb6c14be90 2024-03-25T19:46:58.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.117+0000 7fcb70e42640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.117+0000 7fcb70e42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.117+0000 7fcb70e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.117+0000 7fcb70e42640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:46:58.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.117+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb6c0af940 con 0x7fcb6c14be90 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.128+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159844782 0 0) 0x7fcb6c0b0a70 con 0x7fcb6c0b1730 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.128+0000 7fcb70e42640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.128+0000 7fcb70e42640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.128+0000 7fcb70e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.128+0000 7fcb70e42640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:46:58.129 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.128+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb6c0b0d50 con 0x7fcb6c0b1730 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3391794049 0 0) 0x7fcb6c0af940 con 0x7fcb6c14be90 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 >> v1:172.21.15.76:6789/0 conn(0x7fcb6c0b1730 legacy=0x7fcb6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 >> v1:172.21.15.50:6789/0 conn(0x7fcb6c0b4d60 legacy=0x7fcb6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 10 monclient: _finish_hunting 0 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 1 monclient: found mon.noname-b 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0b1030 con 0x7fcb6c14be90 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 10 monclient: _finish_auth 0 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 10 monclient: _check_auth_tickets 2024-03-25T19:46:58.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcb640019f0 con 0x7fcb6c14be90 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb70e42640 10 monclient: dump: 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:58.137 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:58.138 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb740a8640 5 monclient: authenticate success, global_id 4481 2024-03-25T19:46:58.138 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.135+0000 7fcb740a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb64001b50 con 0x7fcb6c14be90 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb70e42640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb70e42640 1 -- 172.21.15.50:0/1913123599 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcb64001f60 con 0x7fcb6c14be90 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb740a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb740a8640 4 set_mon_vals no callback set 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb70e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb70e42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.172+0000 7fcb70e42640 10 monclient: dump: 2024-03-25T19:46:58.173 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:58.174 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:58.176 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.173+0000 7fcb740a8640 10 monclient: shutdown 2024-03-25T19:46:58.176 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.173+0000 7fcb740a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:58.176 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.173+0000 7fcb740a8640 1 -- 172.21.15.50:0/1913123599 >> v1:172.21.15.50:6790/0 conn(0x7fcb6c14be90 legacy=0x7fcb6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.173+0000 7fcb740a8640 1 -- 172.21.15.50:0/1913123599 shutdown_connections 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.173+0000 7fcb740a8640 1 -- 172.21.15.50:0/1913123599 >> 172.21.15.50:0/1913123599 conn(0x7fcb6c0617a0 msgr2=0x7fcb6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.173+0000 7fcb740a8640 1 -- 172.21.15.50:0/1913123599 shutdown_connections 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.173+0000 7fcb740a8640 1 -- 172.21.15.50:0/1913123599 wait complete. 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 10 monclient: build_initial_monmap 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 10 monclient: monmap: 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:46:58.177 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 1 librados: starting msgr at 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 1 librados: starting objecter 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 1 Processor -- start 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 1 -- start start 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 1 librados: setting wanted keys 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 1 librados: calling monclient init 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.174+0000 7fcb740a8640 10 monclient: init 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6c0b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 10 monclient(hunting): picked mon.noname-b con 0x7fcb6c0b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6c14be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c143220 con 0x7fcb6c14be90 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c1d7870 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.178 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c13d490 con 0x7fcb6c0b1730 2024-03-25T19:46:58.179 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:58.179 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.175+0000 7fcb740a8640 10 monclient(hunting): authenticate will time out at 5193.440181s 2024-03-25T19:46:58.188 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.188+0000 7fcb72645640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcb6c14be90 0x7fcb6c132d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:36236/0 (socket says 172.21.15.50:36236) 2024-03-25T19:46:58.188 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.188+0000 7fcb72645640 1 -- 172.21.15.50:0/1285155931 learned_addr learned my addr 172.21.15.50:0/1285155931 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:46:58.191 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.191+0000 7fcb627fc640 10 client.?.objecter ms_handle_connect 0x7fcb6c0b4d60 2024-03-25T19:46:58.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.191+0000 7fcb627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.192+0000 7fcb627fc640 10 client.?.objecter ms_handle_connect 0x7fcb6c0b1730 2024-03-25T19:46:58.193 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.192+0000 7fcb627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.195 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.195+0000 7fcb627fc640 10 client.?.objecter ms_handle_connect 0x7fcb6c14be90 2024-03-25T19:46:58.196 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.195+0000 7fcb627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:58.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.201+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972621315 0 0) 0x7fcb6c1d7870 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.201+0000 7fcb627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.201+0000 7fcb627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.201+0000 7fcb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.203 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.201+0000 7fcb627fc640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:46:58.203 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.201+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb50004eb0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.206 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248753873 0 0) 0x7fcb6c143220 con 0x7fcb6c14be90 2024-03-25T19:46:58.206 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.206 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.206 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.206 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:46:58.206 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb6c1d7870 con 0x7fcb6c14be90 2024-03-25T19:46:58.206 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995525427 0 0) 0x7fcb6c13d490 con 0x7fcb6c0b1730 2024-03-25T19:46:58.207 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:58.207 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:58.207 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:58.207 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:46:58.207 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.205+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb6c143220 con 0x7fcb6c0b1730 2024-03-25T19:46:58.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.222+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127357045 0 0) 0x7fcb50004eb0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.222+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb6c13d490 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.222+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcb5c0018f0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.222+0000 7fcb627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.222+0000 7fcb627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:58.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.222+0000 7fcb627fc640 10 monclient(hunting): dump: 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:58.224 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.222+0000 7fcb627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.228+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466778052 0 0) 0x7fcb6c1d7870 con 0x7fcb6c14be90 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.228+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb50004eb0 con 0x7fcb6c14be90 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.228+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcb64001f20 con 0x7fcb6c14be90 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.228+0000 7fcb627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.228+0000 7fcb627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.228+0000 7fcb627fc640 10 monclient(hunting): dump: 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:58.229 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:58.230 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:58.230 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:58.230 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2792234757 0 0) 0x7fcb6c13d490 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 >> v1:172.21.15.50:6790/0 conn(0x7fcb6c0b1730 legacy=0x7fcb6c141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 >> v1:172.21.15.50:6789/0 conn(0x7fcb6c14be90 legacy=0x7fcb6c132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 1 monclient: found mon.b 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:58.239 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c1422c0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 5 monclient: authenticate success, global_id 4489 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 10 monclient: _renew_subs 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c140ea0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 10 monclient: _renew_subs 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c1d8680 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.238+0000 7fcb740a8640 1 librados: init done 2024-03-25T19:46:58.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.239+0000 7fcb740a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.239+0000 7fcb740a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.239+0000 7fcb740a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:58.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.239+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c141240 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.521+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb5c003610 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.521+0000 7fcb627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.521+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcb5c003920 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.521+0000 7fcb627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.522+0000 7fcb627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.522+0000 7fcb627fc640 10 monclient: dump: 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:58.523 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.521+0000 7fcb63fff640 4 set_mon_vals no callback set 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.522+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fcb5c0198f0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.523+0000 7fcb71e44640 10 monclient: get_auth_request con 0x7fcb5006df70 auth_method 0 2024-03-25T19:46:58.524 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.523+0000 7fcb627fc640 10 client.4489.objecter ms_handle_connect 0x7fcb5006df70 2024-03-25T19:46:58.528 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fcb5c0535b0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.529 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 10 client.4489.objecter ms_dispatch 0x7fcb6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:58.529 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:58.529 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:58.529 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T19:46:58.529 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb5c04f0c0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.529 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:58.529 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.528+0000 7fcb627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:58.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.774+0000 7fcb740a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:58.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.774+0000 7fcb740a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:58.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.774+0000 7fcb740a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:46:58.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.774+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb6c1d7a00 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.810 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.810+0000 7fcb627fc640 1 -- 172.21.15.50:0/1285155931 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fcb5c050ff0 con 0x7fcb6c0b4d60 2024-03-25T19:46:58.810 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.810+0000 7fcb627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:58.811 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.810+0000 7fcb627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:58.811 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:46:58.811 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 10 librados: watch_flush enter 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 10 librados: watch_flush exit 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 10 client.4489.objecter successfully canceled tick 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 >> v1:172.21.15.76:6800/2572556811 conn(0x7fcb5006df70 legacy=0x7fcb50070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 10 monclient: shutdown 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 >> v1:172.21.15.76:6789/0 conn(0x7fcb6c0b4d60 legacy=0x7fcb6c1d7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 shutdown_connections 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 >> 172.21.15.50:0/1285155931 conn(0x7fcb6c0617a0 msgr2=0x7fcb6c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.812+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 shutdown_connections 2024-03-25T19:46:58.814 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.813+0000 7fcb740a8640 1 -- 172.21.15.50:0/1285155931 wait complete. 2024-03-25T19:46:58.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:46:58.813+0000 7fcb740a8640 1 librados: shutdown 2024-03-25T19:46:58.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:00.216 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:20.722 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.230 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:20.736 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.270 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:20.776 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.282 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:20.788 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.283 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:20.789 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.293 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:20.764 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.302 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:20.773 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.314 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:20.820 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:00.335 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:20.805 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.632 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:05.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.748+0000 7fc2943fb640 10 monclient: get_monmap_and_config 2024-03-25T19:47:05.748 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.748+0000 7fc2943fb640 10 monclient: build_initial_monmap 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.748+0000 7fc2943fb640 10 monclient: monmap: 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:05.749306+0000 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:05.749306+0000 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.749+0000 7fc2943fb640 1 Processor -- start 2024-03-25T19:47:05.749 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.749+0000 7fc2943fb640 1 -- start start 2024-03-25T19:47:05.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.749+0000 7fc2943fb640 10 monclient: init 2024-03-25T19:47:05.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:05.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:05.750 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:05.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:05.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:05.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c142520 con 0x7fc28c0a98f0 2024-03-25T19:47:05.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0b28a0 con 0x7fc28c1430c0 2024-03-25T19:47:05.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0b2a30 con 0x7fc28c13f710 2024-03-25T19:47:05.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:05.751 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.750+0000 7fc2943fb640 10 monclient(hunting): authenticate will time out at 5201.014272s 2024-03-25T19:47:05.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.753+0000 7fc291996640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fc28c13f710 0x7fc28c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53910/0 (socket says 172.21.15.50:53910) 2024-03-25T19:47:05.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.753+0000 7fc291996640 1 -- 172.21.15.50:0/2697860357 learned_addr learned my addr 172.21.15.50:0/2697860357 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:05.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.764+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602139954 0 0) 0x7fc28c0b2a30 con 0x7fc28c13f710 2024-03-25T19:47:05.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.764+0000 7fc291195640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.764+0000 7fc291195640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.764+0000 7fc291195640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.764+0000 7fc291195640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:47:05.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.764+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc278003520 con 0x7fc28c13f710 2024-03-25T19:47:05.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.768+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630358026 0 0) 0x7fc28c0b28a0 con 0x7fc28c1430c0 2024-03-25T19:47:05.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.768+0000 7fc291195640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.768+0000 7fc291195640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.768+0000 7fc291195640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.768+0000 7fc291195640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:47:05.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.768+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28c0b2a30 con 0x7fc28c1430c0 2024-03-25T19:47:05.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492992466 0 0) 0x7fc278003520 con 0x7fc28c13f710 2024-03-25T19:47:05.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 >> v1:172.21.15.76:6789/0 conn(0x7fc28c1430c0 legacy=0x7fc28c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 >> v1:172.21.15.50:6789/0 conn(0x7fc28c0a98f0 legacy=0x7fc28c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 10 monclient: _finish_hunting 0 2024-03-25T19:47:05.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 1 monclient: found mon.noname-b 2024-03-25T19:47:05.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:05.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0b44c0 con 0x7fc28c13f710 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 10 monclient: _finish_auth 0 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 10 monclient: _check_auth_tickets 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc27c001690 con 0x7fc28c13f710 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc2943fb640 5 monclient: authenticate success, global_id 4487 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc2943fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc291195640 10 monclient: dump: 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:05.781 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.782 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:05.782 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:05.782 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:05.782 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:05.782 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:05.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.780+0000 7fc2943fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:05.796 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc27c0017f0 con 0x7fc28c13f710 2024-03-25T19:47:05.796 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc291195640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:05.796 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc291195640 1 -- 172.21.15.50:0/2697860357 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc27c001c00 con 0x7fc28c13f710 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc291195640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc291195640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc291195640 10 monclient: dump: 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:05.797 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:05.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc2943fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:05.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc2943fb640 4 set_mon_vals no callback set 2024-03-25T19:47:05.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc2943fb640 10 monclient: shutdown 2024-03-25T19:47:05.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc2943fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.797+0000 7fc2943fb640 1 -- 172.21.15.50:0/2697860357 >> v1:172.21.15.50:6790/0 conn(0x7fc28c13f710 legacy=0x7fc28c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.798+0000 7fc2943fb640 1 -- 172.21.15.50:0/2697860357 shutdown_connections 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.798+0000 7fc2943fb640 1 -- 172.21.15.50:0/2697860357 >> 172.21.15.50:0/2697860357 conn(0x7fc28c0b7ea0 msgr2=0x7fc28c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.798+0000 7fc2943fb640 1 -- 172.21.15.50:0/2697860357 shutdown_connections 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.798+0000 7fc2943fb640 1 -- 172.21.15.50:0/2697860357 wait complete. 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.798+0000 7fc2943fb640 10 monclient: build_initial_monmap 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.798+0000 7fc2943fb640 10 monclient: monmap: 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:05.800 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.799+0000 7fc2943fb640 1 librados: starting msgr at 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.799+0000 7fc2943fb640 1 librados: starting objecter 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.799+0000 7fc2943fb640 1 Processor -- start 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.799+0000 7fc2943fb640 1 -- start start 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.799+0000 7fc2943fb640 1 librados: setting wanted keys 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.799+0000 7fc2943fb640 1 librados: calling monclient init 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.799+0000 7fc2943fb640 10 monclient: init 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc28c0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc28c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc28c1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:05.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0b1870 con 0x7fc28c0a98f0 2024-03-25T19:47:05.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0ac910 con 0x7fc28c13f710 2024-03-25T19:47:05.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0abb00 con 0x7fc28c1430c0 2024-03-25T19:47:05.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:05.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.800+0000 7fc2943fb640 10 monclient(hunting): authenticate will time out at 5201.063831s 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.810+0000 7fc292998640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fc28c1430c0 0x7fc28c1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53962/0 (socket says 172.21.15.50:53962) 2024-03-25T19:47:05.810 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.810+0000 7fc292998640 1 -- 172.21.15.50:0/682910171 learned_addr learned my addr 172.21.15.50:0/682910171 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.812+0000 7fc282ffd640 10 client.?.objecter ms_handle_connect 0x7fc28c1430c0 2024-03-25T19:47:05.811 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.812+0000 7fc282ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:05.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.816+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147836356 0 0) 0x7fc28c0abb00 con 0x7fc28c1430c0 2024-03-25T19:47:05.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.816+0000 7fc282ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.816+0000 7fc282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.816+0000 7fc282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.816+0000 7fc282ffd640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:47:05.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.816+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc270004eb0 con 0x7fc28c1430c0 2024-03-25T19:47:05.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.817+0000 7fc282ffd640 10 client.?.objecter ms_handle_connect 0x7fc28c13f710 2024-03-25T19:47:05.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.817+0000 7fc282ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:05.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.821+0000 7fc282ffd640 10 client.?.objecter ms_handle_connect 0x7fc28c0a98f0 2024-03-25T19:47:05.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.821+0000 7fc282ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:05.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.828+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385663909 0 0) 0x7fc28c0ac910 con 0x7fc28c13f710 2024-03-25T19:47:05.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.828+0000 7fc282ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.828+0000 7fc282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.828+0000 7fc282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.828+0000 7fc282ffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:47:05.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.828+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28c0abb00 con 0x7fc28c13f710 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.833+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1827873291 0 0) 0x7fc270004eb0 con 0x7fc28c1430c0 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.833+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc28c0ac910 con 0x7fc28c1430c0 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.833+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc284001c90 con 0x7fc28c1430c0 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.833+0000 7fc282ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.833+0000 7fc282ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.833+0000 7fc282ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:05.833 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:05.834 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:05.834 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:05.834 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:05.834 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:05.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.833+0000 7fc282ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:05.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.834+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163848407 0 0) 0x7fc28c142520 con 0x7fc28c0a98f0 2024-03-25T19:47:05.834 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.834+0000 7fc282ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:05.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.834+0000 7fc282ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:05.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.834+0000 7fc282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:05.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.834+0000 7fc282ffd640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:47:05.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.834+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc270004eb0 con 0x7fc28c0a98f0 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 280829212 0 0) 0x7fc28c0ac910 con 0x7fc28c1430c0 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 >> v1:172.21.15.76:6789/0 conn(0x7fc28c13f710 legacy=0x7fc28c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 >> v1:172.21.15.50:6789/0 conn(0x7fc28c0a98f0 legacy=0x7fc28c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 1 monclient: found mon.c 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0abde0 con 0x7fc28c1430c0 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc282ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:05.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 5 monclient: authenticate success, global_id 4490 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 10 monclient: _renew_subs 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c138f40 con 0x7fc28c1430c0 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 10 monclient: _renew_subs 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c1363f0 con 0x7fc28c1430c0 2024-03-25T19:47:05.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.839+0000 7fc2943fb640 1 librados: init done 2024-03-25T19:47:05.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.840+0000 7fc2943fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:05.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.840+0000 7fc2943fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:05.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.840+0000 7fc2943fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:05.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:05.840+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c0627b0 con 0x7fc28c1430c0 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.159+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc284003990 con 0x7fc28c1430c0 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.159+0000 7fc282ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.159+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fc284003d60 con 0x7fc28c1430c0 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.159+0000 7fc282ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.159+0000 7fc282ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.159+0000 7fc282ffd640 10 monclient: dump: 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:06.159 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:06.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.159+0000 7fc290994640 4 set_mon_vals no callback set 2024-03-25T19:47:06.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.160+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fc284003f80 con 0x7fc28c1430c0 2024-03-25T19:47:06.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.160+0000 7fc292197640 10 monclient: get_auth_request con 0x7fc27006de00 auth_method 0 2024-03-25T19:47:06.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.161+0000 7fc282ffd640 10 client.4490.objecter ms_handle_connect 0x7fc27006de00 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fc284054a90 con 0x7fc28c1430c0 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 10 client.4490.objecter ms_dispatch 0x7fc28c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2840505a0 con 0x7fc28c1430c0 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:06.166 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.166+0000 7fc282ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:06.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.402+0000 7fc2943fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.402+0000 7fc2943fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.402+0000 7fc2943fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:06.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.402+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc28c1434a0 con 0x7fc28c1430c0 2024-03-25T19:47:06.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.436+0000 7fc282ffd640 1 -- 172.21.15.50:0/682910171 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fc2840524d0 con 0x7fc28c1430c0 2024-03-25T19:47:06.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.436+0000 7fc282ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:06.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.436+0000 7fc282ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:06.436 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:47:06.436 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.438+0000 7fc2943fb640 10 librados: watch_flush enter 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.438+0000 7fc2943fb640 10 librados: watch_flush exit 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.438+0000 7fc2943fb640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 10 client.4490.objecter successfully canceled tick 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 >> v1:172.21.15.76:6800/2572556811 conn(0x7fc27006de00 legacy=0x7fc2700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 10 monclient: shutdown 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 >> v1:172.21.15.50:6790/0 conn(0x7fc28c1430c0 legacy=0x7fc28c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 shutdown_connections 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 >> 172.21.15.50:0/682910171 conn(0x7fc28c0b7ea0 msgr2=0x7fc28c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:06.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 shutdown_connections 2024-03-25T19:47:06.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 1 -- 172.21.15.50:0/682910171 wait complete. 2024-03-25T19:47:06.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:06.439+0000 7fc2943fb640 1 librados: shutdown 2024-03-25T19:47:06.451 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:13.256 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.374+0000 7f39b6ae6640 10 monclient: get_monmap_and_config 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.374+0000 7f39b6ae6640 10 monclient: build_initial_monmap 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.374+0000 7f39b6ae6640 10 monclient: monmap: 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:13.375228+0000 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:13.375228+0000 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:13.374 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:13.375 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:13.375 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 1 Processor -- start 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 1 -- start start 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient: init 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f39b00a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f39b013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f39b01430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0142520 con 0x7f39b013f710 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b28a0 con 0x7f39b01430c0 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b2a30 con 0x7f39b00a98f0 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:13.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.375+0000 7f39b6ae6640 10 monclient(hunting): authenticate will time out at 5208.639668s 2024-03-25T19:47:13.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.383+0000 7f39b4882640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f39b00a98f0 0x7f39b00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:40858/0 (socket says 172.21.15.50:40858) 2024-03-25T19:47:13.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.383+0000 7f39b4882640 1 -- 172.21.15.50:0/1223690756 learned_addr learned my addr 172.21.15.50:0/1223690756 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:13.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.403+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 866538038 0 0) 0x7f39b00b28a0 con 0x7f39b01430c0 2024-03-25T19:47:13.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.403+0000 7f39af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.403+0000 7f39af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.403+0000 7f39af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.403+0000 7f39af7fe640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:47:13.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.403+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a0003520 con 0x7f39b01430c0 2024-03-25T19:47:13.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.405+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109416852 0 0) 0x7f39b0142520 con 0x7f39b013f710 2024-03-25T19:47:13.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.405+0000 7f39af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.405+0000 7f39af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.405+0000 7f39af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.405+0000 7f39af7fe640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:47:13.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.405+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b00b28a0 con 0x7f39b013f710 2024-03-25T19:47:13.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.412+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302775062 0 0) 0x7f39b00b2a30 con 0x7f39b00a98f0 2024-03-25T19:47:13.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.412+0000 7f39af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.412+0000 7f39af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.412+0000 7f39af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.412+0000 7f39af7fe640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:47:13.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.412+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0142520 con 0x7f39b00a98f0 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988022175 0 0) 0x7f39b00b28a0 con 0x7f39b013f710 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 >> v1:172.21.15.50:6790/0 conn(0x7f39b00a98f0 legacy=0x7f39b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 >> v1:172.21.15.76:6789/0 conn(0x7f39b01430c0 legacy=0x7f39b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 1 monclient: found mon.noname-a 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00b44c0 con 0x7f39b013f710 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:13.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f399c001670 con 0x7f39b013f710 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39b6ae6640 5 monclient: authenticate success, global_id 4518 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39b6ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39af7fe640 10 monclient: dump: 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:13.422 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:13.423 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:13.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.421+0000 7f39b6ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:13.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f399c0017d0 con 0x7f39b013f710 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39af7fe640 1 -- 172.21.15.50:0/1223690756 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f399c001be0 con 0x7f39b013f710 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39af7fe640 10 monclient: dump: 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:13.442 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:13.443 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:13.443 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:13.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39b6ae6640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:13.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.442+0000 7f39b6ae6640 4 set_mon_vals no callback set 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.443+0000 7f39b6ae6640 10 monclient: shutdown 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.443+0000 7f39b6ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.443+0000 7f39b6ae6640 1 -- 172.21.15.50:0/1223690756 >> v1:172.21.15.50:6789/0 conn(0x7f39b013f710 legacy=0x7f39b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.443+0000 7f39b6ae6640 1 -- 172.21.15.50:0/1223690756 shutdown_connections 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.443+0000 7f39b6ae6640 1 -- 172.21.15.50:0/1223690756 >> 172.21.15.50:0/1223690756 conn(0x7f39b00b7ea0 msgr2=0x7f39b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.443+0000 7f39b6ae6640 1 -- 172.21.15.50:0/1223690756 shutdown_connections 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.443+0000 7f39b6ae6640 1 -- 172.21.15.50:0/1223690756 wait complete. 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 10 monclient: build_initial_monmap 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 10 monclient: monmap: 2024-03-25T19:47:13.444 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 1 librados: starting msgr at 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 1 librados: starting objecter 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 1 Processor -- start 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 1 -- start start 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 1 librados: setting wanted keys 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 1 librados: calling monclient init 2024-03-25T19:47:13.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.444+0000 7f39b6ae6640 10 monclient: init 2024-03-25T19:47:13.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f39b00a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f39b013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f39b01430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0139190 con 0x7f39b013f710 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0137590 con 0x7f39b01430c0 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0138b40 con 0x7f39b00a98f0 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:13.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.445+0000 7f39b6ae6640 10 monclient(hunting): authenticate will time out at 5208.709221s 2024-03-25T19:47:13.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.452+0000 7f39b4882640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f39b00a98f0 0x7f39b0136e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:40916/0 (socket says 172.21.15.50:40916) 2024-03-25T19:47:13.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.452+0000 7f39b4882640 1 -- 172.21.15.50:0/696483526 learned_addr learned my addr 172.21.15.50:0/696483526 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:13.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.453+0000 7f39ad7fa640 10 client.?.objecter ms_handle_connect 0x7f39b00a98f0 2024-03-25T19:47:13.453 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.453+0000 7f39ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:13.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.462+0000 7f39ad7fa640 10 client.?.objecter ms_handle_connect 0x7f39b01430c0 2024-03-25T19:47:13.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.462+0000 7f39ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:13.464 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.464+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810758176 0 0) 0x7f39b0138b40 con 0x7f39b00a98f0 2024-03-25T19:47:13.464 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.464+0000 7f39ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.464 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.464+0000 7f39ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.464 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.464+0000 7f39ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.464 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.464+0000 7f39ad7fa640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:47:13.464 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.464+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994004eb0 con 0x7f39b00a98f0 2024-03-25T19:47:13.472 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.472+0000 7f39ad7fa640 10 client.?.objecter ms_handle_connect 0x7f39b013f710 2024-03-25T19:47:13.472 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.472+0000 7f39ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:13.473 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.474+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696958588 0 0) 0x7f39b0137590 con 0x7f39b01430c0 2024-03-25T19:47:13.473 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.474+0000 7f39ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.473 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.474+0000 7f39ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.473 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.474+0000 7f39ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.474 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.474+0000 7f39ad7fa640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:47:13.474 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.474+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0138b40 con 0x7f39b01430c0 2024-03-25T19:47:13.481 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.480+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2868778609 0 0) 0x7f3994004eb0 con 0x7f39b00a98f0 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.480+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39b0137590 con 0x7f39b00a98f0 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.480+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f39a4001910 con 0x7f39b00a98f0 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.480+0000 7f39ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.480+0000 7f39ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.480+0000 7f39ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:13.482 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:13.483 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:13.483 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:13.483 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.480+0000 7f39ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:13.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.488+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2586036385 0 0) 0x7f39b0138b40 con 0x7f39b01430c0 2024-03-25T19:47:13.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.488+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3994004eb0 con 0x7f39b01430c0 2024-03-25T19:47:13.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.488+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f39a8001a70 con 0x7f39b01430c0 2024-03-25T19:47:13.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.488+0000 7f39ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.488+0000 7f39ad7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.488+0000 7f39ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:13.489 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:13.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.489+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403806337 0 0) 0x7f39b0139190 con 0x7f39b013f710 2024-03-25T19:47:13.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.489+0000 7f39ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:13.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.489+0000 7f39ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:13.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.489+0000 7f39ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:13.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.489+0000 7f39ad7fa640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:47:13.490 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.489+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0138b40 con 0x7f39b013f710 2024-03-25T19:47:13.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1215146741 0 0) 0x7f3994004eb0 con 0x7f39b01430c0 2024-03-25T19:47:13.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 >> v1:172.21.15.50:6790/0 conn(0x7f39b00a98f0 legacy=0x7f39b0136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 >> v1:172.21.15.50:6789/0 conn(0x7f39b013f710 legacy=0x7f39b01364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:13.499 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:13.500 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 1 monclient: found mon.b 2024-03-25T19:47:13.500 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:13.500 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:13.500 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b0136bf0 con 0x7f39b01430c0 2024-03-25T19:47:13.500 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 10 monclient: _renew_subs 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b013afe0 con 0x7f39b01430c0 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 10 monclient: _renew_subs 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b0139620 con 0x7f39b01430c0 2024-03-25T19:47:13.501 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.499+0000 7f39b6ae6640 1 librados: init done 2024-03-25T19:47:13.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.500+0000 7f39b6ae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:13.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.500+0000 7f39b6ae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:13.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.500+0000 7f39b6ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:13.502 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.500+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b0138f60 con 0x7f39b01430c0 2024-03-25T19:47:13.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.785+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a8001f60 con 0x7f39b01430c0 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.785+0000 7f39ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.785+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f39a8003990 con 0x7f39b01430c0 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.785+0000 7f39ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.785+0000 7f39ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.785+0000 7f39aeffd640 4 set_mon_vals no callback set 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.785+0000 7f39ad7fa640 10 monclient: dump: 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:13.785 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:13.786 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:13.786 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:13.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.786+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f39a8003bb0 con 0x7f39b01430c0 2024-03-25T19:47:13.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.786+0000 7f39b4882640 10 monclient: get_auth_request con 0x7f399406df20 auth_method 0 2024-03-25T19:47:13.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.786+0000 7f39ad7fa640 10 client.4501.objecter ms_handle_connect 0x7f399406df20 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.792+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f39a8053640 con 0x7f39b01430c0 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.792+0000 7f39ad7fa640 10 client.4501.objecter ms_dispatch 0x7f39b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.792+0000 7f39ad7fa640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.792+0000 7f39ad7fa640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.792+0000 7f39ad7fa640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.793+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39a804f150 con 0x7f39b01430c0 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.793+0000 7f39ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:13.792 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:13.793+0000 7f39ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:14.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.048+0000 7f39b6ae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.048+0000 7f39b6ae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.048+0000 7f39b6ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:14.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.048+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39b00627b0 con 0x7f39b01430c0 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.073+0000 7f39ad7fa640 1 -- 172.21.15.50:0/696483526 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f39a8051080 con 0x7f39b01430c0 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.073+0000 7f39ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.073+0000 7f39ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:47:14.073 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.075+0000 7f39b6ae6640 10 librados: watch_flush enter 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.075+0000 7f39b6ae6640 10 librados: watch_flush exit 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.075+0000 7f39b6ae6640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.075+0000 7f39b6ae6640 10 client.4501.objecter successfully canceled tick 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 >> v1:172.21.15.76:6800/2572556811 conn(0x7f399406df20 legacy=0x7f39940703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 10 monclient: shutdown 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 >> v1:172.21.15.76:6789/0 conn(0x7f39b01430c0 legacy=0x7f39b01e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 shutdown_connections 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 >> 172.21.15.50:0/696483526 conn(0x7f39b00b7ea0 msgr2=0x7f39b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:14.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 shutdown_connections 2024-03-25T19:47:14.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 1 -- 172.21.15.50:0/696483526 wait complete. 2024-03-25T19:47:14.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:14.076+0000 7f39b6ae6640 1 librados: shutdown 2024-03-25T19:47:14.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:20.239 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:40.745 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.291 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:40.797 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.333 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:06:40.839 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.351 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:40.821 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.355 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:40.825 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.368 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:40.874 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.375 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:40.880 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.383 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:06:40.854 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.397 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:06:40.903 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:20.895 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.010+0000 7fedee427640 10 monclient: get_monmap_and_config 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.010+0000 7fedee427640 10 monclient: build_initial_monmap 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.010+0000 7fedee427640 10 monclient: monmap: 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:21.011383+0000 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:21.011383+0000 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:21.010 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:21.011 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:21.011 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.011+0000 7fedee427640 1 Processor -- start 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.011+0000 7fedee427640 1 -- start start 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.011+0000 7fedee427640 10 monclient: init 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 10 monclient(hunting): picked mon.noname-a con 0x7fede80b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 10 monclient(hunting): picked mon.noname-b con 0x7fede80b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 10 monclient(hunting): picked mon.noname-c con 0x7fede814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede80af940 con 0x7fede80b4d60 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede80b0a70 con 0x7fede814be90 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede80b0d50 con 0x7fede80b1730 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:21.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.012+0000 7fedee427640 10 monclient(hunting): authenticate will time out at 5216.276098s 2024-03-25T19:47:21.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.015+0000 7fede77fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fede80b1730 0x7fede80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:52960/0 (socket says 172.21.15.50:52960) 2024-03-25T19:47:21.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.015+0000 7fede77fe640 1 -- 172.21.15.50:0/3131826366 learned_addr learned my addr 172.21.15.50:0/3131826366 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:21.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.019+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044106129 0 0) 0x7fede80b0d50 con 0x7fede80b1730 2024-03-25T19:47:21.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.019+0000 7fede6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.019+0000 7fede6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.019+0000 7fede6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.019+0000 7fede6ffd640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:47:21.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.019+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd0003520 con 0x7fede80b1730 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674421756 0 0) 0x7fedd0003520 con 0x7fede80b1730 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 >> v1:172.21.15.76:6789/0 conn(0x7fede814be90 legacy=0x7fede814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 >> v1:172.21.15.50:6789/0 conn(0x7fede80b4d60 legacy=0x7fede813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 1 monclient: found mon.noname-b 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:47:21.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede80b1030 con 0x7fede80b1730 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feddc001690 con 0x7fede80b1730 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fedee427640 5 monclient: authenticate success, global_id 4499 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fedee427640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fede6ffd640 10 monclient: dump: 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.032 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.033 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:21.033 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:21.033 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:21.033 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:21.033 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:21.033 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.031+0000 7fedee427640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feddc0017f0 con 0x7fede80b1730 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fede6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fede6ffd640 1 -- 172.21.15.50:0/3131826366 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feddc001c00 con 0x7fede80b1730 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fede6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fede6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fede6ffd640 10 monclient: dump: 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:21.039 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fedee427640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:21.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.039+0000 7fedee427640 4 set_mon_vals no callback set 2024-03-25T19:47:21.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.040+0000 7fedee427640 10 monclient: shutdown 2024-03-25T19:47:21.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.040+0000 7fedee427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:21.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.040+0000 7fedee427640 1 -- 172.21.15.50:0/3131826366 >> v1:172.21.15.50:6790/0 conn(0x7fede80b1730 legacy=0x7fede80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.040+0000 7fedee427640 1 -- 172.21.15.50:0/3131826366 shutdown_connections 2024-03-25T19:47:21.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.040+0000 7fedee427640 1 -- 172.21.15.50:0/3131826366 >> 172.21.15.50:0/3131826366 conn(0x7fede80617a0 msgr2=0x7fede8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:21.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.040+0000 7fedee427640 1 -- 172.21.15.50:0/3131826366 shutdown_connections 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.041+0000 7fedee427640 1 -- 172.21.15.50:0/3131826366 wait complete. 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.041+0000 7fedee427640 10 monclient: build_initial_monmap 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.041+0000 7fedee427640 10 monclient: monmap: 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:21.042 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:21.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.042+0000 7fedee427640 1 librados: starting msgr at 2024-03-25T19:47:21.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.042+0000 7fedee427640 1 librados: starting objecter 2024-03-25T19:47:21.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.042+0000 7fedee427640 1 Processor -- start 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.042+0000 7fedee427640 1 -- start start 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.042+0000 7fedee427640 1 librados: setting wanted keys 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.042+0000 7fedee427640 1 librados: calling monclient init 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.042+0000 7fedee427640 10 monclient: init 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 10 monclient(hunting): picked mon.noname-a con 0x7fede80b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 10 monclient(hunting): picked mon.noname-b con 0x7fede80b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 10 monclient(hunting): picked mon.noname-c con 0x7fede814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede8141bf0 con 0x7fede80b1730 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede8141ed0 con 0x7fede80b4d60 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fede81421b0 con 0x7fede814be90 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:21.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.043+0000 7fedee427640 10 monclient(hunting): authenticate will time out at 5216.307068s 2024-03-25T19:47:21.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.057+0000 7fede77fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fede80b4d60 0x7fede8133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:54808/0 (socket says 172.21.15.50:54808) 2024-03-25T19:47:21.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.057+0000 7fede77fe640 1 -- 172.21.15.50:0/2685739478 learned_addr learned my addr 172.21.15.50:0/2685739478 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:21.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.058+0000 7fede4ff9640 10 client.?.objecter ms_handle_connect 0x7fede80b4d60 2024-03-25T19:47:21.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.058+0000 7fede4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.060+0000 7fede4ff9640 10 client.?.objecter ms_handle_connect 0x7fede814be90 2024-03-25T19:47:21.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.060+0000 7fede4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.069+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952345115 0 0) 0x7fede81421b0 con 0x7fede814be90 2024-03-25T19:47:21.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.069+0000 7fede4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.069+0000 7fede4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.069+0000 7fede4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.069+0000 7fede4ff9640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:47:21.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.070+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd4004eb0 con 0x7fede814be90 2024-03-25T19:47:21.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.071+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061356016 0 0) 0x7fede8141ed0 con 0x7fede80b4d60 2024-03-25T19:47:21.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.071+0000 7fede4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.071+0000 7fede4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.071+0000 7fede4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.071+0000 7fede4ff9640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:47:21.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.071+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede81421b0 con 0x7fede80b4d60 2024-03-25T19:47:21.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.072+0000 7fede4ff9640 10 client.?.objecter ms_handle_connect 0x7fede80b1730 2024-03-25T19:47:21.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.072+0000 7fede4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.083+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783482244 0 0) 0x7fedd4004eb0 con 0x7fede814be90 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.083+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fede8141ed0 con 0x7fede814be90 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.083+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fede0001b90 con 0x7fede814be90 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.083+0000 7fede4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.083+0000 7fede4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.083+0000 7fede4ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:21.083 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:21.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.083+0000 7fede4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:21.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.086+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408966598 0 0) 0x7fede8141bf0 con 0x7fede80b1730 2024-03-25T19:47:21.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.086+0000 7fede4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:21.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.086+0000 7fede4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:21.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.086+0000 7fede4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:21.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.086+0000 7fede4ff9640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:47:21.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.086+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd4004eb0 con 0x7fede80b1730 2024-03-25T19:47:21.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 73853902 0 0) 0x7fede8141ed0 con 0x7fede814be90 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 >> v1:172.21.15.76:6789/0 conn(0x7fede80b4d60 legacy=0x7fede8133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 >> v1:172.21.15.50:6789/0 conn(0x7fede80b1730 legacy=0x7fede81425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 1 monclient: found mon.c 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede8140f00 con 0x7fede814be90 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fede4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:21.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 5 monclient: authenticate success, global_id 4502 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 10 monclient: _renew_subs 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede81406f0 con 0x7fede814be90 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 10 monclient: _renew_subs 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fede8140380 con 0x7fede814be90 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.093+0000 7fedee427640 1 librados: init done 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.094+0000 7fedee427640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.094+0000 7fedee427640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.094+0000 7fedee427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:21.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.094+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede81401f0 con 0x7fede814be90 2024-03-25T19:47:21.394 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.394+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fede0002080 con 0x7fede814be90 2024-03-25T19:47:21.394 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.394+0000 7fede4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.394+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fede0003b30 con 0x7fede814be90 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.394+0000 7fede4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.394+0000 7fede4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.394+0000 7fede4ff9640 10 monclient: dump: 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:21.395 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:21.396 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:21.396 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:21.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.394+0000 7fede67fc640 4 set_mon_vals no callback set 2024-03-25T19:47:21.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.395+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fede0003d50 con 0x7fede814be90 2024-03-25T19:47:21.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.396+0000 7fede7fff640 10 monclient: get_auth_request con 0x7fedd406ddb0 auth_method 0 2024-03-25T19:47:21.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.396+0000 7fede4ff9640 10 client.4502.objecter ms_handle_connect 0x7fedd406ddb0 2024-03-25T19:47:21.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fede00537f0 con 0x7fede814be90 2024-03-25T19:47:21.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 10 client.4502.objecter ms_dispatch 0x7fede8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:21.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:21.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:21.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-25T19:47:21.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fede004f300 con 0x7fede814be90 2024-03-25T19:47:21.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:21.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.401+0000 7fede4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:21.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.618+0000 7fedee427640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:21.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.618+0000 7fedee427640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:21.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.618+0000 7fedee427640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:21.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.618+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fede80627b0 con 0x7fede814be90 2024-03-25T19:47:21.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.656+0000 7fede4ff9640 1 -- 172.21.15.50:0/2685739478 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fede0051230 con 0x7fede814be90 2024-03-25T19:47:21.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.656+0000 7fede4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:21.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.656+0000 7fede4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:21.656 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:47:21.656 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.658+0000 7fedee427640 10 librados: watch_flush enter 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.658+0000 7fedee427640 10 librados: watch_flush exit 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.658+0000 7fedee427640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.658+0000 7fedee427640 10 client.4502.objecter successfully canceled tick 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 >> v1:172.21.15.76:6800/2572556811 conn(0x7fedd406ddb0 legacy=0x7fedd4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 10 monclient: shutdown 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 >> v1:172.21.15.50:6790/0 conn(0x7fede814be90 legacy=0x7fede81341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 shutdown_connections 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 >> 172.21.15.50:0/2685739478 conn(0x7fede80617a0 msgr2=0x7fede813c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 shutdown_connections 2024-03-25T19:47:21.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 1 -- 172.21.15.50:0/2685739478 wait complete. 2024-03-25T19:47:21.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:21.659+0000 7fedee427640 1 librados: shutdown 2024-03-25T19:47:21.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:28.477 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:28.597 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.598+0000 7f1c68d32640 10 monclient: get_monmap_and_config 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.598+0000 7f1c68d32640 10 monclient: build_initial_monmap 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.598+0000 7f1c68d32640 10 monclient: monmap: 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:28.599024+0000 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:28.599024+0000 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:28.598 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:28.599 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:28.599 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:28.599 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.598+0000 7f1c68d32640 1 Processor -- start 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.598+0000 7f1c68d32640 1 -- start start 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.598+0000 7f1c68d32640 10 monclient: init 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 10 monclient(hunting): picked mon.noname-a con 0x7f1c640a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6413f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 10 monclient(hunting): picked mon.noname-c con 0x7f1c641430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:28.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c64142520 con 0x7f1c640a98f0 2024-03-25T19:47:28.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640b28a0 con 0x7f1c641430c0 2024-03-25T19:47:28.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640b2a30 con 0x7f1c6413f710 2024-03-25T19:47:28.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:28.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.599+0000 7f1c68d32640 10 monclient(hunting): authenticate will time out at 5223.863503s 2024-03-25T19:47:28.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.612+0000 7f1c6259c640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1c640a98f0 0x7f1c640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:56282/0 (socket says 172.21.15.50:56282) 2024-03-25T19:47:28.611 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.612+0000 7f1c6259c640 1 -- 172.21.15.50:0/4168926594 learned_addr learned my addr 172.21.15.50:0/4168926594 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:28.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.626+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139404034 0 0) 0x7f1c64142520 con 0x7f1c640a98f0 2024-03-25T19:47:28.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.626+0000 7f1c6159a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.626+0000 7f1c6159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.626+0000 7f1c6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.626+0000 7f1c6159a640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:47:28.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.626+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c003520 con 0x7f1c640a98f0 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.631+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308792139 0 0) 0x7f1c640b28a0 con 0x7f1c641430c0 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.631+0000 7f1c6159a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.631+0000 7f1c6159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.631+0000 7f1c6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.631+0000 7f1c6159a640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.631+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c64142520 con 0x7f1c641430c0 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.633+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192129187 0 0) 0x7f1c640b2a30 con 0x7f1c6413f710 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.633+0000 7f1c6159a640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.633+0000 7f1c6159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.633+0000 7f1c6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.633+0000 7f1c6159a640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:47:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.633+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c640b28a0 con 0x7f1c6413f710 2024-03-25T19:47:28.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.636+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1246502575 0 0) 0x7f1c4c003520 con 0x7f1c640a98f0 2024-03-25T19:47:28.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.636+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 >> v1:172.21.15.50:6790/0 conn(0x7f1c6413f710 legacy=0x7f1c641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.636+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 >> v1:172.21.15.76:6789/0 conn(0x7f1c641430c0 legacy=0x7f1c6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 10 monclient: _finish_hunting 0 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 1 monclient: found mon.noname-a 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640b44c0 con 0x7f1c640a98f0 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 10 monclient: _finish_auth 0 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 10 monclient: _check_auth_tickets 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c68d32640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:47:28.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.637+0000 7f1c68d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:28.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.640+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1c54001650 con 0x7f1c640a98f0 2024-03-25T19:47:28.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.640+0000 7f1c6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:28.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.640+0000 7f1c6159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:28.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.640+0000 7f1c6159a640 10 monclient: dump: 2024-03-25T19:47:28.640 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:28.640 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:28.640 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:28.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.640+0000 7f1c68d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:28.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c540018a0 con 0x7f1c640a98f0 2024-03-25T19:47:28.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c6159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:28.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c6159a640 1 -- 172.21.15.50:0/4168926594 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1c54001c30 con 0x7f1c640a98f0 2024-03-25T19:47:28.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c68d32640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:28.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c68d32640 4 set_mon_vals no callback set 2024-03-25T19:47:28.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:28.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c6159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.651+0000 7f1c6159a640 10 monclient: dump: 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:28.652 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.652+0000 7f1c68d32640 10 monclient: shutdown 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.652+0000 7f1c68d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.652+0000 7f1c68d32640 1 -- 172.21.15.50:0/4168926594 >> v1:172.21.15.50:6789/0 conn(0x7f1c640a98f0 legacy=0x7f1c640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.652+0000 7f1c68d32640 1 -- 172.21.15.50:0/4168926594 shutdown_connections 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.652+0000 7f1c68d32640 1 -- 172.21.15.50:0/4168926594 >> 172.21.15.50:0/4168926594 conn(0x7f1c640b7ea0 msgr2=0x7f1c640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.653+0000 7f1c68d32640 1 -- 172.21.15.50:0/4168926594 shutdown_connections 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.653+0000 7f1c68d32640 1 -- 172.21.15.50:0/4168926594 wait complete. 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 10 monclient: build_initial_monmap 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 10 monclient: monmap: 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:28.654 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 1 librados: starting msgr at 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 1 librados: starting objecter 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 1 Processor -- start 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 1 -- start start 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 1 librados: setting wanted keys 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 1 librados: calling monclient init 2024-03-25T19:47:28.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.654+0000 7f1c68d32640 10 monclient: init 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 10 monclient(hunting): picked mon.noname-a con 0x7f1c640a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 10 monclient(hunting): picked mon.noname-c con 0x7f1c641430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c641db110 con 0x7f1c640a98f0 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c64138d90 con 0x7f1c6413f710 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c641376a0 con 0x7f1c641430c0 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:28.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.655+0000 7f1c68d32640 10 monclient(hunting): authenticate will time out at 5223.919159s 2024-03-25T19:47:28.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.667+0000 7f1c61d9b640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1c6413f710 0x7f1c641daa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:34058/0 (socket says 172.21.15.50:34058) 2024-03-25T19:47:28.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.667+0000 7f1c61d9b640 1 -- 172.21.15.50:0/2147476156 learned_addr learned my addr 172.21.15.50:0/2147476156 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:28.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.669+0000 7f1c52ffd640 10 client.?.objecter ms_handle_connect 0x7f1c6413f710 2024-03-25T19:47:28.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.669+0000 7f1c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:28.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.678+0000 7f1c52ffd640 10 client.?.objecter ms_handle_connect 0x7f1c640a98f0 2024-03-25T19:47:28.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.678+0000 7f1c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:28.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.678+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907399994 0 0) 0x7f1c64138d90 con 0x7f1c6413f710 2024-03-25T19:47:28.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.678+0000 7f1c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.678+0000 7f1c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.678+0000 7f1c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.678+0000 7f1c52ffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:47:28.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.679+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c44004eb0 con 0x7f1c6413f710 2024-03-25T19:47:28.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.680+0000 7f1c52ffd640 10 client.?.objecter ms_handle_connect 0x7f1c641430c0 2024-03-25T19:47:28.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.680+0000 7f1c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:28.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.687+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250305073 0 0) 0x7f1c641376a0 con 0x7f1c641430c0 2024-03-25T19:47:28.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.687+0000 7f1c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.687+0000 7f1c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.687+0000 7f1c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.687+0000 7f1c52ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:47:28.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.687+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c64138d90 con 0x7f1c641430c0 2024-03-25T19:47:28.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.698+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276617609 0 0) 0x7f1c641db110 con 0x7f1c640a98f0 2024-03-25T19:47:28.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.698+0000 7f1c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:28.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.698+0000 7f1c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:28.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.698+0000 7f1c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:28.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.698+0000 7f1c52ffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:47:28.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.698+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c641376a0 con 0x7f1c640a98f0 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.702+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1816277292 0 0) 0x7f1c44004eb0 con 0x7f1c6413f710 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.702+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c641db110 con 0x7f1c6413f710 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.702+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1c58001a10 con 0x7f1c6413f710 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.702+0000 7f1c52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.702+0000 7f1c52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.702+0000 7f1c52ffd640 10 monclient(hunting): dump: 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:28.702 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:28.703 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:28.703 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:28.703 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:28.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.702+0000 7f1c52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:28.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2449618001 0 0) 0x7f1c641db110 con 0x7f1c6413f710 2024-03-25T19:47:28.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 >> v1:172.21.15.50:6790/0 conn(0x7f1c641430c0 legacy=0x7f1c64132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 >> v1:172.21.15.50:6789/0 conn(0x7f1c640a98f0 legacy=0x7f1c641369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 1 monclient: found mon.b 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640b7260 con 0x7f1c6413f710 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 5 monclient: authenticate success, global_id 4513 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 10 monclient: _renew_subs 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:28.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c640b74b0 con 0x7f1c6413f710 2024-03-25T19:47:28.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:28.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 10 monclient: _renew_subs 2024-03-25T19:47:28.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:28.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c641337f0 con 0x7f1c6413f710 2024-03-25T19:47:28.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.718+0000 7f1c68d32640 1 librados: init done 2024-03-25T19:47:28.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.719+0000 7f1c68d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:28.720 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.719+0000 7f1c68d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:28.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.719+0000 7f1c68d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:28.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.719+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c64137830 con 0x7f1c6413f710 2024-03-25T19:47:28.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.994+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c58001c70 con 0x7f1c6413f710 2024-03-25T19:47:28.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.994+0000 7f1c52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:28.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.994+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1c58003a60 con 0x7f1c6413f710 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.994+0000 7f1c52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.994+0000 7f1c52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.994+0000 7f1c52ffd640 10 monclient: dump: 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:28.995 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.994+0000 7f1c60d99640 4 set_mon_vals no callback set 2024-03-25T19:47:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.995+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f1c58003c80 con 0x7f1c6413f710 2024-03-25T19:47:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.996+0000 7f1c6259c640 10 monclient: get_auth_request con 0x7f1c4406ddf0 auth_method 0 2024-03-25T19:47:28.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:28.996+0000 7f1c52ffd640 10 client.4513.objecter ms_handle_connect 0x7f1c4406ddf0 2024-03-25T19:47:29.000 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.001+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f1c580548b0 con 0x7f1c6413f710 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.001+0000 7f1c52ffd640 10 client.4513.objecter ms_dispatch 0x7f1c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.001+0000 7f1c52ffd640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.001+0000 7f1c52ffd640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.001+0000 7f1c52ffd640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.002+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c580503c0 con 0x7f1c6413f710 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.002+0000 7f1c52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:29.001 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.002+0000 7f1c52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:29.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.235+0000 7f1c68d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.235+0000 7f1c68d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.235+0000 7f1c68d32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:47:29.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.235+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c64133a40 con 0x7f1c6413f710 2024-03-25T19:47:29.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.329+0000 7f1c52ffd640 1 -- 172.21.15.50:0/2147476156 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f1c580522f0 con 0x7f1c6413f710 2024-03-25T19:47:29.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.329+0000 7f1c52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:29.329 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.329+0000 7f1c52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:29.329 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:47:29.330 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 10 librados: watch_flush enter 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 10 librados: watch_flush exit 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 10 client.4513.objecter successfully canceled tick 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 >> v1:172.21.15.76:6800/2572556811 conn(0x7f1c4406ddf0 legacy=0x7f1c440702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 10 monclient: shutdown 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 >> v1:172.21.15.76:6789/0 conn(0x7f1c6413f710 legacy=0x7f1c641daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:29.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 shutdown_connections 2024-03-25T19:47:29.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 >> 172.21.15.50:0/2147476156 conn(0x7f1c640b7ea0 msgr2=0x7f1c64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:29.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 shutdown_connections 2024-03-25T19:47:29.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.331+0000 7f1c68d32640 1 -- 172.21.15.50:0/2147476156 wait complete. 2024-03-25T19:47:29.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:29.332+0000 7f1c68d32640 1 librados: shutdown 2024-03-25T19:47:29.344 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:36.150 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.262+0000 7fbb6df02640 10 monclient: get_monmap_and_config 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.262+0000 7fbb6df02640 10 monclient: build_initial_monmap 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.262+0000 7fbb6df02640 10 monclient: monmap: 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:36.263221+0000 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:36.263221+0000 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:36.262 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:36.263 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:36.263 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:36.263 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:36.263 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:36.264 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 1 Processor -- start 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 1 -- start start 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient: init 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient(hunting): picked mon.noname-b con 0x7fbb680a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient(hunting): picked mon.noname-a con 0x7fbb6813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient(hunting): picked mon.noname-c con 0x7fbb681430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb68142520 con 0x7fbb6813f710 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb680b28a0 con 0x7fbb681430c0 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb680b2a30 con 0x7fbb680a98f0 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:36.265 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.263+0000 7fbb6df02640 10 monclient(hunting): authenticate will time out at 5231.527638s 2024-03-25T19:47:36.267 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.268+0000 7fbb67fff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbb681430c0 0x7fbb6814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:38556/0 (socket says 172.21.15.50:38556) 2024-03-25T19:47:36.267 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.268+0000 7fbb67fff640 1 -- 172.21.15.50:0/1562774888 learned_addr learned my addr 172.21.15.50:0/1562774888 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:36.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.282+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773200269 0 0) 0x7fbb680b28a0 con 0x7fbb681430c0 2024-03-25T19:47:36.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.282+0000 7fbb667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.282+0000 7fbb667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.282+0000 7fbb667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.282+0000 7fbb667fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:47:36.282 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.282+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003520 con 0x7fbb681430c0 2024-03-25T19:47:36.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.287+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452882046 0 0) 0x7fbb68142520 con 0x7fbb6813f710 2024-03-25T19:47:36.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.287+0000 7fbb667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.287+0000 7fbb667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.287+0000 7fbb667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.287+0000 7fbb667fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:47:36.288 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.287+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb680b28a0 con 0x7fbb6813f710 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 495609769 0 0) 0x7fbb50003520 con 0x7fbb681430c0 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 >> v1:172.21.15.50:6790/0 conn(0x7fbb680a98f0 legacy=0x7fbb680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 >> v1:172.21.15.50:6789/0 conn(0x7fbb6813f710 legacy=0x7fbb681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 1 monclient: found mon.noname-c 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb680b44c0 con 0x7fbb681430c0 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:36.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbb60001a10 con 0x7fbb681430c0 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb667fc640 10 monclient: dump: 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:36.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb6df02640 5 monclient: authenticate success, global_id 4516 2024-03-25T19:47:36.298 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.296+0000 7fbb6df02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:36.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb60001b70 con 0x7fbb681430c0 2024-03-25T19:47:36.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:36.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb667fc640 1 -- 172.21.15.50:0/1562774888 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbb60001f80 con 0x7fbb681430c0 2024-03-25T19:47:36.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb6df02640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:36.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb6df02640 4 set_mon_vals no callback set 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.310+0000 7fbb667fc640 10 monclient: dump: 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:36.311 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:36.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.311+0000 7fbb6df02640 10 monclient: shutdown 2024-03-25T19:47:36.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.311+0000 7fbb6df02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:36.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.311+0000 7fbb6df02640 1 -- 172.21.15.50:0/1562774888 >> v1:172.21.15.76:6789/0 conn(0x7fbb681430c0 legacy=0x7fbb6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.311+0000 7fbb6df02640 1 -- 172.21.15.50:0/1562774888 shutdown_connections 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.311+0000 7fbb6df02640 1 -- 172.21.15.50:0/1562774888 >> 172.21.15.50:0/1562774888 conn(0x7fbb680b7ea0 msgr2=0x7fbb680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.311+0000 7fbb6df02640 1 -- 172.21.15.50:0/1562774888 shutdown_connections 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.311+0000 7fbb6df02640 1 -- 172.21.15.50:0/1562774888 wait complete. 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 10 monclient: build_initial_monmap 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 10 monclient: monmap: 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:36.314 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 1 librados: starting msgr at 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 1 librados: starting objecter 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 1 Processor -- start 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 1 -- start start 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 1 librados: setting wanted keys 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 1 librados: calling monclient init 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.312+0000 7fbb6df02640 10 monclient: init 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 10 monclient(hunting): picked mon.noname-b con 0x7fbb680a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:36.315 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 10 monclient(hunting): picked mon.noname-a con 0x7fbb6813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:36.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 10 monclient(hunting): picked mon.noname-c con 0x7fbb681430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:36.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb68137190 con 0x7fbb6813f710 2024-03-25T19:47:36.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb681d41e0 con 0x7fbb680a98f0 2024-03-25T19:47:36.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb681d4430 con 0x7fbb681430c0 2024-03-25T19:47:36.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:36.316 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.313+0000 7fbb6df02640 10 monclient(hunting): authenticate will time out at 5231.577233s 2024-03-25T19:47:36.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.325+0000 7fbb67fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fbb681430c0 0x7fbb68130ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:43524/0 (socket says 172.21.15.50:43524) 2024-03-25T19:47:36.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.325+0000 7fbb67fff640 1 -- 172.21.15.50:0/401735047 learned_addr learned my addr 172.21.15.50:0/401735047 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:36.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.332+0000 7fbb4ffff640 10 client.?.objecter ms_handle_connect 0x7fbb681430c0 2024-03-25T19:47:36.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.332+0000 7fbb4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:36.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.332+0000 7fbb4ffff640 10 client.?.objecter ms_handle_connect 0x7fbb680a98f0 2024-03-25T19:47:36.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.332+0000 7fbb4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:36.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.337+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726661536 0 0) 0x7fbb681d4430 con 0x7fbb681430c0 2024-03-25T19:47:36.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.337+0000 7fbb4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.337+0000 7fbb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.337+0000 7fbb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.337+0000 7fbb4ffff640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:47:36.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.337+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb681430c0 2024-03-25T19:47:36.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.340+0000 7fbb4ffff640 10 client.?.objecter ms_handle_connect 0x7fbb6813f710 2024-03-25T19:47:36.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.340+0000 7fbb4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:36.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.343+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934265333 0 0) 0x7fbb681d41e0 con 0x7fbb680a98f0 2024-03-25T19:47:36.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.344+0000 7fbb4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.344+0000 7fbb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.344+0000 7fbb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.344+0000 7fbb4ffff640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:47:36.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.344+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb681d4430 con 0x7fbb680a98f0 2024-03-25T19:47:36.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.347+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3193842535 0 0) 0x7fbb54004eb0 con 0x7fbb681430c0 2024-03-25T19:47:36.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.347+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb681d41e0 con 0x7fbb681430c0 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.347+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbb60001f40 con 0x7fbb681430c0 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.347+0000 7fbb4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.347+0000 7fbb4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.347+0000 7fbb4ffff640 10 monclient(hunting): dump: 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:36.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.347+0000 7fbb4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:36.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.353+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735244879 0 0) 0x7fbb68137190 con 0x7fbb6813f710 2024-03-25T19:47:36.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.353+0000 7fbb4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:36.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.353+0000 7fbb4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:36.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.353+0000 7fbb4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:36.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.353+0000 7fbb4ffff640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:47:36.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.353+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb54004eb0 con 0x7fbb6813f710 2024-03-25T19:47:36.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 634905695 0 0) 0x7fbb681d41e0 con 0x7fbb681430c0 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 >> v1:172.21.15.76:6789/0 conn(0x7fbb680a98f0 legacy=0x7fbb681d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 >> v1:172.21.15.50:6789/0 conn(0x7fbb6813f710 legacy=0x7fbb681352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 1 monclient: found mon.c 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6813bf90 con 0x7fbb681430c0 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 10 monclient: _finish_auth 0 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 5 monclient: authenticate success, global_id 4514 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 10 monclient: _renew_subs 2024-03-25T19:47:36.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:36.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb681d5060 con 0x7fbb681430c0 2024-03-25T19:47:36.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:36.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 10 monclient: _renew_subs 2024-03-25T19:47:36.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:36.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb68139f00 con 0x7fbb681430c0 2024-03-25T19:47:36.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.360+0000 7fbb6df02640 1 librados: init done 2024-03-25T19:47:36.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.361+0000 7fbb6df02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:36.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.361+0000 7fbb6df02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:36.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.361+0000 7fbb6df02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:36.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.361+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb680627b0 con 0x7fbb681430c0 2024-03-25T19:47:36.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.686+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb60004e10 con 0x7fbb681430c0 2024-03-25T19:47:36.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.686+0000 7fbb4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:36.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.686+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbb60005120 con 0x7fbb681430c0 2024-03-25T19:47:36.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.686+0000 7fbb4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.686+0000 7fbb4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.686+0000 7fbb4ffff640 10 monclient: dump: 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:36.687 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.686+0000 7fbb65ffb640 4 set_mon_vals no callback set 2024-03-25T19:47:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.687+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fbb6001b0f0 con 0x7fbb681430c0 2024-03-25T19:47:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.688+0000 7fbb677fe640 10 monclient: get_auth_request con 0x7fbb5406de00 auth_method 0 2024-03-25T19:47:36.688 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.688+0000 7fbb4ffff640 10 client.4514.objecter ms_handle_connect 0x7fbb5406de00 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fbb60054bf0 con 0x7fbb681430c0 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 10 client.4514.objecter ms_dispatch 0x7fbb68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb60050530 con 0x7fbb681430c0 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:36.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.695+0000 7fbb4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:36.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.941+0000 7fbb6df02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:36.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.941+0000 7fbb6df02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:36.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.941+0000 7fbb6df02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:36.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:36.941+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb68069f40 con 0x7fbb681430c0 2024-03-25T19:47:37.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.072+0000 7fbb4ffff640 1 -- 172.21.15.50:0/401735047 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fbb60052630 con 0x7fbb681430c0 2024-03-25T19:47:37.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.072+0000 7fbb4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:37.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.072+0000 7fbb4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:37.073 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:47:37.073 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.074+0000 7fbb6df02640 10 librados: watch_flush enter 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.074+0000 7fbb6df02640 10 librados: watch_flush exit 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.074+0000 7fbb6df02640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.074+0000 7fbb6df02640 10 client.4514.objecter successfully canceled tick 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 >> v1:172.21.15.76:6800/2572556811 conn(0x7fbb5406de00 legacy=0x7fbb540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 10 monclient: shutdown 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 >> v1:172.21.15.50:6790/0 conn(0x7fbb681430c0 legacy=0x7fbb68130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:37.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 shutdown_connections 2024-03-25T19:47:37.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 >> 172.21.15.50:0/401735047 conn(0x7fbb680b7ea0 msgr2=0x7fbb68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:37.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 shutdown_connections 2024-03-25T19:47:37.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 1 -- 172.21.15.50:0/401735047 wait complete. 2024-03-25T19:47:37.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:37.075+0000 7fbb6df02640 1 librados: shutdown 2024-03-25T19:47:37.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:40.339 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:00.845 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.350 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:00.856 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.386 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:00.892 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.398 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:00.869 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.411 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:00.882 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.420 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:00.926 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.429 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:00.935 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.439 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:00.910 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:40.453 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:00.959 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:43.893 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:44.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.002+0000 7f4435f7b640 10 monclient: get_monmap_and_config 2024-03-25T19:47:44.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.002+0000 7f4435f7b640 10 monclient: build_initial_monmap 2024-03-25T19:47:44.003 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.002+0000 7f4435f7b640 10 monclient: monmap: 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:44.004625+0000 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:44.004625+0000 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:44.004 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:44.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.003+0000 7f4435f7b640 1 Processor -- start 2024-03-25T19:47:44.005 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.003+0000 7f4435f7b640 1 -- start start 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.003+0000 7f4435f7b640 10 monclient: init 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f44300b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f44300b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f443014be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300af940 con 0x7f44300b4d60 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300b0a70 con 0x7f44300b1730 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300b0d50 con 0x7f443014be90 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4435f7b640 10 monclient(hunting): authenticate will time out at 5239.268941s 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4426ffd640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f44300b1730 0x7f44300b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:46706/0 (socket says 172.21.15.50:46706) 2024-03-25T19:47:44.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.004+0000 7f4426ffd640 1 -- 172.21.15.50:0/533118626 learned_addr learned my addr 172.21.15.50:0/533118626 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:44.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.054+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251503519 0 0) 0x7f44300b0d50 con 0x7f443014be90 2024-03-25T19:47:44.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.054+0000 7f442effd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.054+0000 7f442effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.054+0000 7f442effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.054+0000 7f442effd640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:47:44.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.054+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f441c003520 con 0x7f443014be90 2024-03-25T19:47:44.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.062+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608357099 0 0) 0x7f44300af940 con 0x7f44300b4d60 2024-03-25T19:47:44.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.062+0000 7f442effd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.062+0000 7f442effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.062+0000 7f442effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.062+0000 7f442effd640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:47:44.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.062+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44300b0d50 con 0x7f44300b4d60 2024-03-25T19:47:44.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127168790 0 0) 0x7f441c003520 con 0x7f443014be90 2024-03-25T19:47:44.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 >> v1:172.21.15.76:6789/0 conn(0x7f44300b1730 legacy=0x7f44300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 >> v1:172.21.15.50:6789/0 conn(0x7f44300b4d60 legacy=0x7f443013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:44.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 1 monclient: found mon.noname-b 2024-03-25T19:47:44.077 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44300b1030 con 0x7f443014be90 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 10 monclient: _finish_auth 0 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4428001a10 con 0x7f443014be90 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f442effd640 10 monclient: dump: 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.078 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f4435f7b640 5 monclient: authenticate success, global_id 4517 2024-03-25T19:47:44.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.076+0000 7f4435f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4428001b70 con 0x7f443014be90 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f442effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f442effd640 1 -- 172.21.15.50:0/533118626 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4428001f80 con 0x7f443014be90 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f4435f7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f4435f7b640 4 set_mon_vals no callback set 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f442effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f442effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.084+0000 7f442effd640 10 monclient: dump: 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:44.085 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.086 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.086 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.086 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:44.086 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:44.086 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:44.086 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:44.086 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:44.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.085+0000 7f4435f7b640 10 monclient: shutdown 2024-03-25T19:47:44.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.085+0000 7f4435f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:44.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.085+0000 7f4435f7b640 1 -- 172.21.15.50:0/533118626 >> v1:172.21.15.50:6790/0 conn(0x7f443014be90 legacy=0x7f443014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.085+0000 7f4435f7b640 1 -- 172.21.15.50:0/533118626 shutdown_connections 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.085+0000 7f4435f7b640 1 -- 172.21.15.50:0/533118626 >> 172.21.15.50:0/533118626 conn(0x7f44300617a0 msgr2=0x7f4430061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.085+0000 7f4435f7b640 1 -- 172.21.15.50:0/533118626 shutdown_connections 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.085+0000 7f4435f7b640 1 -- 172.21.15.50:0/533118626 wait complete. 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.086+0000 7f4435f7b640 10 monclient: build_initial_monmap 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.086+0000 7f4435f7b640 10 monclient: monmap: 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:44.089 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.086+0000 7f4435f7b640 1 librados: starting msgr at 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.086+0000 7f4435f7b640 1 librados: starting objecter 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.086+0000 7f4435f7b640 1 Processor -- start 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 1 -- start start 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 1 librados: setting wanted keys 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 1 librados: calling monclient init 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient: init 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f44300b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:44.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f44300b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:44.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f443014be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:44.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44301384c0 con 0x7f44300b1730 2024-03-25T19:47:44.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44301db920 con 0x7f44300b4d60 2024-03-25T19:47:44.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44301dbab0 con 0x7f443014be90 2024-03-25T19:47:44.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:44.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.087+0000 7f4435f7b640 10 monclient(hunting): authenticate will time out at 5239.352341s 2024-03-25T19:47:44.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.096+0000 7f442ffff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f443014be90 0x7f443012fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:54424/0 (socket says 172.21.15.50:54424) 2024-03-25T19:47:44.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.096+0000 7f442ffff640 1 -- 172.21.15.50:0/3459274466 learned_addr learned my addr 172.21.15.50:0/3459274466 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:44.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.098+0000 7f442cff9640 10 client.?.objecter ms_handle_connect 0x7f443014be90 2024-03-25T19:47:44.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.098+0000 7f442cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.101+0000 7f442cff9640 10 client.?.objecter ms_handle_connect 0x7f44300b1730 2024-03-25T19:47:44.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.101+0000 7f442cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.107 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.107+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23360302 0 0) 0x7f44301dbab0 con 0x7f443014be90 2024-03-25T19:47:44.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.107+0000 7f442cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.107+0000 7f442cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.107+0000 7f442cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.107+0000 7f442cff9640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:47:44.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.107+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4410004eb0 con 0x7f443014be90 2024-03-25T19:47:44.110 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.109+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235038853 0 0) 0x7f44301384c0 con 0x7f44300b1730 2024-03-25T19:47:44.110 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.109+0000 7f442cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.110 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.109+0000 7f442cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.110 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.109+0000 7f442cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.110 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.109+0000 7f442cff9640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:47:44.110 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.109+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44301dbab0 con 0x7f44300b1730 2024-03-25T19:47:44.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.117+0000 7f442cff9640 10 client.?.objecter ms_handle_connect 0x7f44300b4d60 2024-03-25T19:47:44.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.117+0000 7f442cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:44.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.123+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 320770501 0 0) 0x7f4410004eb0 con 0x7f443014be90 2024-03-25T19:47:44.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.123+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44301384c0 con 0x7f443014be90 2024-03-25T19:47:44.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.123+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4428001f80 con 0x7f443014be90 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.123+0000 7f442cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.123+0000 7f442cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.123+0000 7f442cff9640 10 monclient(hunting): dump: 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:44.124 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:44.125 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:44.125 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:44.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.123+0000 7f442cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346668636 0 0) 0x7f44301db920 con 0x7f44300b4d60 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4410004eb0 con 0x7f44300b4d60 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2189916367 0 0) 0x7f44301dbab0 con 0x7f44300b1730 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44301db920 con 0x7f44300b1730 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4420001930 con 0x7f44300b1730 2024-03-25T19:47:44.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:44.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.124+0000 7f442cff9640 10 monclient(hunting): dump: 2024-03-25T19:47:44.127 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:44.127 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:44.127 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.127 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.127 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.128 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:44.128 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:44.128 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:44.128 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:44.128 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:44.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3812577297 0 0) 0x7f44301384c0 con 0x7f443014be90 2024-03-25T19:47:44.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 >> v1:172.21.15.76:6789/0 conn(0x7f44300b4d60 legacy=0x7f4430136a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 >> v1:172.21.15.50:6789/0 conn(0x7f44300b1730 legacy=0x7f44301db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:44.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 1 monclient: found mon.c 2024-03-25T19:47:44.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:44.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:44.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430135e60 con 0x7f443014be90 2024-03-25T19:47:44.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:44.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:44.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f442cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:44.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 5 monclient: authenticate success, global_id 4520 2024-03-25T19:47:44.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 10 monclient: _renew_subs 2024-03-25T19:47:44.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:44.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4430137110 con 0x7f443014be90 2024-03-25T19:47:44.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:44.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 10 monclient: _renew_subs 2024-03-25T19:47:44.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:44.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4430143ae0 con 0x7f443014be90 2024-03-25T19:47:44.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.132+0000 7f4435f7b640 1 librados: init done 2024-03-25T19:47:44.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.133+0000 7f4435f7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.133+0000 7f4435f7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.138 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.133+0000 7f4435f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:44.138 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.133+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44301dab90 con 0x7f443014be90 2024-03-25T19:47:44.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.443+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4428003c20 con 0x7f443014be90 2024-03-25T19:47:44.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.443+0000 7f442cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.444+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4428003e50 con 0x7f443014be90 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.444+0000 7f442cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.444+0000 7f442cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.444+0000 7f442cff9640 10 monclient: dump: 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:44.445 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:44.446 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:44.446 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:44.446 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:44.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.444+0000 7f442e7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:44.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.444+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f4428004070 con 0x7f443014be90 2024-03-25T19:47:44.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.445+0000 7f442f7fe640 10 monclient: get_auth_request con 0x7f441006dfb0 auth_method 0 2024-03-25T19:47:44.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.445+0000 7f442cff9640 10 client.4520.objecter ms_handle_connect 0x7f441006dfb0 2024-03-25T19:47:44.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f4428054980 con 0x7f443014be90 2024-03-25T19:47:44.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 10 client.4520.objecter ms_dispatch 0x7f4430000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:44.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:44.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:44.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T19:47:44.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4428050490 con 0x7f443014be90 2024-03-25T19:47:44.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:44.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.450+0000 7f442cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:44.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.708+0000 7f4435f7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:44.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.708+0000 7f4435f7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:44.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.708+0000 7f4435f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:44.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.708+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44300627b0 con 0x7f443014be90 2024-03-25T19:47:44.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.736+0000 7f442cff9640 1 -- 172.21.15.50:0/3459274466 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f44280523c0 con 0x7f443014be90 2024-03-25T19:47:44.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.736+0000 7f442cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:44.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.736+0000 7f442cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:44.737 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:47:44.737 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:44.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 10 librados: watch_flush enter 2024-03-25T19:47:44.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 10 librados: watch_flush exit 2024-03-25T19:47:44.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T19:47:44.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 10 client.4520.objecter successfully canceled tick 2024-03-25T19:47:44.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 >> v1:172.21.15.76:6800/2572556811 conn(0x7f441006dfb0 legacy=0x7f4410070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 10 monclient: shutdown 2024-03-25T19:47:44.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:44.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 >> v1:172.21.15.50:6790/0 conn(0x7f443014be90 legacy=0x7f443012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:44.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 shutdown_connections 2024-03-25T19:47:44.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 >> 172.21.15.50:0/3459274466 conn(0x7f44300617a0 msgr2=0x7f443013c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:44.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 shutdown_connections 2024-03-25T19:47:44.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 1 -- 172.21.15.50:0/3459274466 wait complete. 2024-03-25T19:47:44.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:44.738+0000 7f4435f7b640 1 librados: shutdown 2024-03-25T19:47:44.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:51.557 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:51.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.676+0000 7f52c3fbc640 10 monclient: get_monmap_and_config 2024-03-25T19:47:51.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.676+0000 7f52c3fbc640 10 monclient: build_initial_monmap 2024-03-25T19:47:51.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.676+0000 7f52c3fbc640 10 monclient: monmap: 2024-03-25T19:47:51.676 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:51.676 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:51.676 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:51.677399+0000 2024-03-25T19:47:51.676 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:51.677399+0000 2024-03-25T19:47:51.677 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:51.677 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:51.677 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:51.677 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:51.677 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:51.677 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.677+0000 7f52c3fbc640 1 Processor -- start 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.677+0000 7f52c3fbc640 1 -- start start 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.677+0000 7f52c3fbc640 10 monclient: init 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 10 monclient(hunting): picked mon.noname-b con 0x7f52bc0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 10 monclient(hunting): picked mon.noname-c con 0x7f52bc13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 10 monclient(hunting): picked mon.noname-a con 0x7f52bc1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc142520 con 0x7f52bc1430c0 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc0b28a0 con 0x7f52bc13f710 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc0b2a30 con 0x7f52bc0a98f0 2024-03-25T19:47:51.678 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:51.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.678+0000 7f52c3fbc640 10 monclient(hunting): authenticate will time out at 5246.942113s 2024-03-25T19:47:51.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.682+0000 7f52c1d58640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f52bc0a98f0 0x7f52bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:40208/0 (socket says 172.21.15.50:40208) 2024-03-25T19:47:51.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.682+0000 7f52c1d58640 1 -- 172.21.15.50:0/767629664 learned_addr learned my addr 172.21.15.50:0/767629664 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:51.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.702+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395502955 0 0) 0x7f52bc0b2a30 con 0x7f52bc0a98f0 2024-03-25T19:47:51.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.702+0000 7f52c0d56640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.702+0000 7f52c0d56640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.702+0000 7f52c0d56640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.702+0000 7f52c0d56640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:47:51.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.702+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a8003520 con 0x7f52bc0a98f0 2024-03-25T19:47:51.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.705+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648053012 0 0) 0x7f52bc0b28a0 con 0x7f52bc13f710 2024-03-25T19:47:51.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.705+0000 7f52c0d56640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.705+0000 7f52c0d56640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.705+0000 7f52c0d56640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.705+0000 7f52c0d56640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:47:51.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.705+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52bc0b2a30 con 0x7f52bc13f710 2024-03-25T19:47:51.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.706+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54671111 0 0) 0x7f52bc142520 con 0x7f52bc1430c0 2024-03-25T19:47:51.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.706+0000 7f52c0d56640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.706+0000 7f52c0d56640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.706+0000 7f52c0d56640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.706+0000 7f52c0d56640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:47:51.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.706+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52bc0b28a0 con 0x7f52bc1430c0 2024-03-25T19:47:51.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3364051402 0 0) 0x7f52bc0b28a0 con 0x7f52bc1430c0 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 >> v1:172.21.15.50:6790/0 conn(0x7f52bc0a98f0 legacy=0x7f52bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 >> v1:172.21.15.76:6789/0 conn(0x7f52bc13f710 legacy=0x7f52bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 10 monclient: _finish_hunting 0 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 1 monclient: found mon.noname-a 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc0b44c0 con 0x7f52bc1430c0 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 10 monclient: _finish_auth 0 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 10 monclient: _check_auth_tickets 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f52b4001930 con 0x7f52bc1430c0 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c3fbc640 5 monclient: authenticate success, global_id 4548 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c3fbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:51.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c0d56640 10 monclient: dump: 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:51.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.713+0000 7f52c3fbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:51.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52b4001a90 con 0x7f52bc1430c0 2024-03-25T19:47:51.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c0d56640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:51.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c0d56640 1 -- 172.21.15.50:0/767629664 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f52b4001ee0 con 0x7f52bc1430c0 2024-03-25T19:47:51.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c3fbc640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:51.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c3fbc640 4 set_mon_vals no callback set 2024-03-25T19:47:51.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c0d56640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:51.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c0d56640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.741+0000 7f52c0d56640 10 monclient: dump: 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:51.742 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.742+0000 7f52c3fbc640 10 monclient: shutdown 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.742+0000 7f52c3fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.742+0000 7f52c3fbc640 1 -- 172.21.15.50:0/767629664 >> v1:172.21.15.50:6789/0 conn(0x7f52bc1430c0 legacy=0x7f52bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.742+0000 7f52c3fbc640 1 -- 172.21.15.50:0/767629664 shutdown_connections 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.742+0000 7f52c3fbc640 1 -- 172.21.15.50:0/767629664 >> 172.21.15.50:0/767629664 conn(0x7f52bc0b7ea0 msgr2=0x7f52bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.743+0000 7f52c3fbc640 1 -- 172.21.15.50:0/767629664 shutdown_connections 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.743+0000 7f52c3fbc640 1 -- 172.21.15.50:0/767629664 wait complete. 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 10 monclient: build_initial_monmap 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 10 monclient: monmap: 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:51.744 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 1 librados: starting msgr at 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 1 librados: starting objecter 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 1 Processor -- start 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 1 -- start start 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 1 librados: setting wanted keys 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 1 librados: calling monclient init 2024-03-25T19:47:51.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.744+0000 7f52c3fbc640 10 monclient: init 2024-03-25T19:47:51.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:51.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:51.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 10 monclient(hunting): picked mon.noname-c con 0x7f52bc0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:51.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 10 monclient(hunting): picked mon.noname-b con 0x7f52bc13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:51.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 10 monclient(hunting): picked mon.noname-a con 0x7f52bc1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:51.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc139990 con 0x7f52bc1430c0 2024-03-25T19:47:51.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc1d51f0 con 0x7f52bc13f710 2024-03-25T19:47:51.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52bc137a00 con 0x7f52bc0a98f0 2024-03-25T19:47:51.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:51.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.745+0000 7f52c3fbc640 10 monclient(hunting): authenticate will time out at 5247.009310s 2024-03-25T19:47:51.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.754+0000 7f52c2559640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f52bc1430c0 0x7f52bc1db410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34794/0 (socket says 172.21.15.50:34794) 2024-03-25T19:47:51.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.754+0000 7f52c2559640 1 -- 172.21.15.50:0/3824830524 learned_addr learned my addr 172.21.15.50:0/3824830524 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:51.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.755+0000 7f52b27fc640 10 client.?.objecter ms_handle_connect 0x7f52bc0a98f0 2024-03-25T19:47:51.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.755+0000 7f52b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:51.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.757+0000 7f52b27fc640 10 client.?.objecter ms_handle_connect 0x7f52bc1430c0 2024-03-25T19:47:51.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.757+0000 7f52b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:51.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.763+0000 7f52b27fc640 10 client.?.objecter ms_handle_connect 0x7f52bc13f710 2024-03-25T19:47:51.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.763+0000 7f52b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:51.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.765+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644895422 0 0) 0x7f52bc137a00 con 0x7f52bc0a98f0 2024-03-25T19:47:51.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.765+0000 7f52b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.765+0000 7f52b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.765+0000 7f52b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.765+0000 7f52b27fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:47:51.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.765+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0004eb0 con 0x7f52bc0a98f0 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.775+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3213921014 0 0) 0x7f52a0004eb0 con 0x7f52bc0a98f0 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.775+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52bc137a00 con 0x7f52bc0a98f0 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.775+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f52b8001950 con 0x7f52bc0a98f0 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.775+0000 7f52b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.775+0000 7f52b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.775+0000 7f52b27fc640 10 monclient(hunting): dump: 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:51.775 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:51.776 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:51.776 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:51.776 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:51.776 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:51.776 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:51.776 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:51.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.775+0000 7f52b27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:51.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.779+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832910583 0 0) 0x7f52bc1d51f0 con 0x7f52bc13f710 2024-03-25T19:47:51.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.779+0000 7f52b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.779+0000 7f52b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.779 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.779+0000 7f52b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.779 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.779+0000 7f52b27fc640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:47:51.779 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.779+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0004eb0 con 0x7f52bc13f710 2024-03-25T19:47:51.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.785+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280490678 0 0) 0x7f52bc139990 con 0x7f52bc1430c0 2024-03-25T19:47:51.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.785+0000 7f52b27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:51.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.785+0000 7f52b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:51.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.785+0000 7f52b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:51.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.785+0000 7f52b27fc640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:47:51.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.785+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52bc1d51f0 con 0x7f52bc1430c0 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.786+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1700302786 0 0) 0x7f52bc137a00 con 0x7f52bc0a98f0 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.786+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 >> v1:172.21.15.76:6789/0 conn(0x7f52bc13f710 legacy=0x7f52bc1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 >> v1:172.21.15.50:6789/0 conn(0x7f52bc1430c0 legacy=0x7f52bc1db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 1 monclient: found mon.c 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc138630 con 0x7f52bc0a98f0 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:51.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 10 monclient: _renew_subs 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52bc1e2dd0 con 0x7f52bc0a98f0 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 10 monclient: _renew_subs 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52bc1e3140 con 0x7f52bc0a98f0 2024-03-25T19:47:51.788 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.787+0000 7f52c3fbc640 1 librados: init done 2024-03-25T19:47:51.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.788+0000 7f52c3fbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:51.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.788+0000 7f52c3fbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:51.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.788+0000 7f52c3fbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:51.789 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:51.788+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52bc0627b0 con 0x7f52bc0a98f0 2024-03-25T19:47:52.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52b80047e0 con 0x7f52bc0a98f0 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f52b8004af0 con 0x7f52bc0a98f0 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b27fc640 10 monclient: dump: 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:52.100 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:52.101 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:52.101 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:52.101 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:52.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b3fff640 4 set_mon_vals no callback set 2024-03-25T19:47:52.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.100+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f52b801aac0 con 0x7f52bc0a98f0 2024-03-25T19:47:52.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.101+0000 7f52c1557640 10 monclient: get_auth_request con 0x7f52a006de80 auth_method 0 2024-03-25T19:47:52.101 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.101+0000 7f52b27fc640 10 client.4526.objecter ms_handle_connect 0x7f52a006de80 2024-03-25T19:47:52.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f52b8054740 con 0x7f52bc0a98f0 2024-03-25T19:47:52.111 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 10 client.4526.objecter ms_dispatch 0x7f52bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:52.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:52.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:52.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T19:47:52.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52b8050080 con 0x7f52bc0a98f0 2024-03-25T19:47:52.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:52.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.112+0000 7f52b27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:52.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.365+0000 7f52c3fbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.365+0000 7f52c3fbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.365+0000 7f52c3fbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:47:52.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.365+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52bc1387c0 con 0x7f52bc0a98f0 2024-03-25T19:47:52.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.400+0000 7f52b27fc640 1 -- 172.21.15.50:0/3824830524 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f52b8052180 con 0x7f52bc0a98f0 2024-03-25T19:47:52.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.400+0000 7f52b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:52.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.400+0000 7f52b27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:52.400 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:47:52.400 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.402+0000 7f52c3fbc640 10 librados: watch_flush enter 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.402+0000 7f52c3fbc640 10 librados: watch_flush exit 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.402+0000 7f52c3fbc640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.402+0000 7f52c3fbc640 10 client.4526.objecter successfully canceled tick 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 >> v1:172.21.15.76:6800/2572556811 conn(0x7f52a006de80 legacy=0x7f52a0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 10 monclient: shutdown 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:52.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 >> v1:172.21.15.50:6790/0 conn(0x7f52bc0a98f0 legacy=0x7f52bc137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:52.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 shutdown_connections 2024-03-25T19:47:52.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 >> 172.21.15.50:0/3824830524 conn(0x7f52bc0b7ea0 msgr2=0x7f52bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:52.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 shutdown_connections 2024-03-25T19:47:52.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 1 -- 172.21.15.50:0/3824830524 wait complete. 2024-03-25T19:47:52.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:52.403+0000 7f52c3fbc640 1 librados: shutdown 2024-03-25T19:47:52.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:59.221 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:59.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.338+0000 7f6582546640 10 monclient: get_monmap_and_config 2024-03-25T19:47:59.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.338+0000 7f6582546640 10 monclient: build_initial_monmap 2024-03-25T19:47:59.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.338+0000 7f6582546640 10 monclient: monmap: 2024-03-25T19:47:59.338 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:59.338 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:59.338 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:47:59.339438+0000 2024-03-25T19:47:59.339 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:47:59.339438+0000 2024-03-25T19:47:59.339 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:59.339 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:59.339 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:59.339 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:47:59.339 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:47:59.339 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.339+0000 7f6582546640 1 Processor -- start 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.339+0000 7f6582546640 1 -- start start 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.339+0000 7f6582546640 10 monclient: init 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 10 monclient(hunting): picked mon.noname-c con 0x7f657c0a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 10 monclient(hunting): picked mon.noname-a con 0x7f657c13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:59.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 10 monclient(hunting): picked mon.noname-b con 0x7f657c1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:59.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c142520 con 0x7f657c13f710 2024-03-25T19:47:59.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c0b28a0 con 0x7f657c0a98f0 2024-03-25T19:47:59.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c0b2a30 con 0x7f657c1430c0 2024-03-25T19:47:59.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:59.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.340+0000 7f6582546640 10 monclient(hunting): authenticate will time out at 5254.604284s 2024-03-25T19:47:59.344 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.344+0000 7f6580ae3640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f657c1430c0 0x7f657c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:34948/0 (socket says 172.21.15.50:34948) 2024-03-25T19:47:59.344 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.344+0000 7f6580ae3640 1 -- 172.21.15.50:0/1038348100 learned_addr learned my addr 172.21.15.50:0/1038348100 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:59.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124039533 0 0) 0x7f657c0b2a30 con 0x7f657c1430c0 2024-03-25T19:47:59.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:47:59.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6564003520 con 0x7f657c1430c0 2024-03-25T19:47:59.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465679356 0 0) 0x7f657c0b28a0 con 0x7f657c0a98f0 2024-03-25T19:47:59.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:47:59.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.358+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f657c0b2a30 con 0x7f657c0a98f0 2024-03-25T19:47:59.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.367+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3758140357 0 0) 0x7f6564003520 con 0x7f657c1430c0 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.367+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 >> v1:172.21.15.76:6789/0 conn(0x7f657c0a98f0 legacy=0x7f657c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.367+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 >> v1:172.21.15.50:6789/0 conn(0x7f657c13f710 legacy=0x7f657c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.367+0000 7f657affd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.367+0000 7f657affd640 1 monclient: found mon.noname-b 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c0b44c0 con 0x7f657c1430c0 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 10 monclient: _finish_auth 0 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f656c001690 con 0x7f657c1430c0 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f657affd640 10 monclient: dump: 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:59.368 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f6582546640 5 monclient: authenticate success, global_id 4529 2024-03-25T19:47:59.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.368+0000 7f6582546640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f656c0017f0 con 0x7f657c1430c0 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f657affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f657affd640 1 -- 172.21.15.50:0/1038348100 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f656c001c00 con 0x7f657c1430c0 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f657affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f657affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f657affd640 10 monclient: dump: 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.384 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.385 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:59.385 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:59.385 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:59.385 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:59.385 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:59.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f6582546640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:59.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.384+0000 7f6582546640 4 set_mon_vals no callback set 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.385+0000 7f6582546640 10 monclient: shutdown 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.385+0000 7f6582546640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.385+0000 7f6582546640 1 -- 172.21.15.50:0/1038348100 >> v1:172.21.15.50:6790/0 conn(0x7f657c1430c0 legacy=0x7f657c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.385+0000 7f6582546640 1 -- 172.21.15.50:0/1038348100 shutdown_connections 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.385+0000 7f6582546640 1 -- 172.21.15.50:0/1038348100 >> 172.21.15.50:0/1038348100 conn(0x7f657c0b7ea0 msgr2=0x7f657c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.385+0000 7f6582546640 1 -- 172.21.15.50:0/1038348100 shutdown_connections 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.385+0000 7f6582546640 1 -- 172.21.15.50:0/1038348100 wait complete. 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 10 monclient: build_initial_monmap 2024-03-25T19:47:59.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 10 monclient: monmap: 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 1 librados: starting msgr at 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 1 librados: starting objecter 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 1 Processor -- start 2024-03-25T19:47:59.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 1 -- start start 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 1 librados: setting wanted keys 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 1 librados: calling monclient init 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.386+0000 7f6582546640 10 monclient: init 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 10 monclient(hunting): picked mon.noname-a con 0x7f657c0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 10 monclient(hunting): picked mon.noname-b con 0x7f657c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 10 monclient(hunting): picked mon.noname-c con 0x7f657c1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c137740 con 0x7f657c0a98f0 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c136c50 con 0x7f657c13f710 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c138fc0 con 0x7f657c1430c0 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:59.389 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.387+0000 7f6582546640 10 monclient(hunting): authenticate will time out at 5254.651178s 2024-03-25T19:47:59.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.398+0000 7f657bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f657c0a98f0 0x7f657c137030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40052/0 (socket says 172.21.15.50:40052) 2024-03-25T19:47:59.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.399+0000 7f657bfff640 1 -- 172.21.15.50:0/3926097034 learned_addr learned my addr 172.21.15.50:0/3926097034 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:47:59.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.401+0000 7f6578ff9640 10 client.?.objecter ms_handle_connect 0x7f657c0a98f0 2024-03-25T19:47:59.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.401+0000 7f6578ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.404+0000 7f6578ff9640 10 client.?.objecter ms_handle_connect 0x7f657c13f710 2024-03-25T19:47:59.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.404+0000 7f6578ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.405+0000 7f6578ff9640 10 client.?.objecter ms_handle_connect 0x7f657c1430c0 2024-03-25T19:47:59.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.405+0000 7f6578ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:59.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.412+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734452187 0 0) 0x7f657c137740 con 0x7f657c0a98f0 2024-03-25T19:47:59.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.412+0000 7f6578ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.412+0000 7f6578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.412+0000 7f6578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.412+0000 7f6578ff9640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:47:59.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.412+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6568004eb0 con 0x7f657c0a98f0 2024-03-25T19:47:59.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.415+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229134508 0 0) 0x7f657c138fc0 con 0x7f657c1430c0 2024-03-25T19:47:59.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.415+0000 7f6578ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.415+0000 7f6578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.415+0000 7f6578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.415+0000 7f6578ff9640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:47:59.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.415+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f657c137740 con 0x7f657c1430c0 2024-03-25T19:47:59.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.425+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498339707 0 0) 0x7f657c136c50 con 0x7f657c13f710 2024-03-25T19:47:59.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.425+0000 7f6578ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:59.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.425+0000 7f6578ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:59.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.425+0000 7f6578ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:59.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.425+0000 7f6578ff9640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:47:59.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.425+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f657c138fc0 con 0x7f657c13f710 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.437+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 123650657 0 0) 0x7f6568004eb0 con 0x7f657c0a98f0 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.437+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f657c136c50 con 0x7f657c0a98f0 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.437+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6574001d90 con 0x7f657c0a98f0 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.437+0000 7f6578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.437+0000 7f6578ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.437+0000 7f6578ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:59.437 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:59.438 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:59.438 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:59.438 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.437+0000 7f6578ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:59.438 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.439+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118616349 0 0) 0x7f657c138fc0 con 0x7f657c13f710 2024-03-25T19:47:59.438 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.439+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6568004eb0 con 0x7f657c13f710 2024-03-25T19:47:59.438 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.439+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6570001870 con 0x7f657c13f710 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.439+0000 7f6578ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.439+0000 7f6578ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.439+0000 7f6578ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:59.439 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.450+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2214480091 0 0) 0x7f657c136c50 con 0x7f657c0a98f0 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 >> v1:172.21.15.50:6790/0 conn(0x7f657c1430c0 legacy=0x7f657c1e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 >> v1:172.21.15.76:6789/0 conn(0x7f657c13f710 legacy=0x7f657c136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 1 monclient: found mon.a 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c139150 con 0x7f657c0a98f0 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6578ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 5 monclient: authenticate success, global_id 4557 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 10 monclient: _renew_subs 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:47:59.451 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657c13b2e0 con 0x7f657c0a98f0 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 10 monclient: _renew_subs 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c1395e0 con 0x7f657c0a98f0 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.451+0000 7f6582546640 1 librados: init done 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.452+0000 7f6582546640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.452+0000 7f6582546640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.452+0000 7f6582546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:47:59.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.452+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c136de0 con 0x7f657c0a98f0 2024-03-25T19:47:59.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.769+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6574001fd0 con 0x7f657c0a98f0 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.769+0000 7f6578ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.769+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6574003de0 con 0x7f657c0a98f0 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.769+0000 7f6578ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.769+0000 7f6578ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.769+0000 7f6578ff9640 10 monclient: dump: 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:47:59.770 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:47:59.771 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:47:59.771 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:47:59.771 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:47:59.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.769+0000 7f657a7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:59.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.770+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f6574004000 con 0x7f657c0a98f0 2024-03-25T19:47:59.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.771+0000 7f657b7fe640 10 monclient: get_auth_request con 0x7f656806df20 auth_method 0 2024-03-25T19:47:59.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.771+0000 7f6578ff9640 10 client.4557.objecter ms_handle_connect 0x7f656806df20 2024-03-25T19:47:59.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f6574054c50 con 0x7f657c0a98f0 2024-03-25T19:47:59.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 10 client.4557.objecter ms_dispatch 0x7f657c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:59.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:59.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:59.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-25T19:47:59.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6574050760 con 0x7f657c0a98f0 2024-03-25T19:47:59.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:59.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:47:59.776+0000 7f6578ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:00.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.035+0000 7f6582546640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.036+0000 7f6582546640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.036+0000 7f6582546640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:00.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.036+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f657c0627b0 con 0x7f657c0a98f0 2024-03-25T19:48:00.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.070+0000 7f6578ff9640 1 -- 172.21.15.50:0/3926097034 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f6574052690 con 0x7f657c0a98f0 2024-03-25T19:48:00.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.070+0000 7f6578ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:00.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.070+0000 7f6578ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:00.070 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:00.070 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:00.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 10 librados: watch_flush enter 2024-03-25T19:48:00.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 10 librados: watch_flush exit 2024-03-25T19:48:00.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-25T19:48:00.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 10 client.4557.objecter successfully canceled tick 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 >> v1:172.21.15.76:6800/2572556811 conn(0x7f656806df20 legacy=0x7f65680703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 10 monclient: shutdown 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 >> v1:172.21.15.50:6789/0 conn(0x7f657c0a98f0 legacy=0x7f657c137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 shutdown_connections 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 >> 172.21.15.50:0/3926097034 conn(0x7f657c0b7ea0 msgr2=0x7f657c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 shutdown_connections 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 1 -- 172.21.15.50:0/3926097034 wait complete. 2024-03-25T19:48:00.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:00.073+0000 7f6582546640 1 librados: shutdown 2024-03-25T19:48:00.085 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:00.401 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:20.907 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.411 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:20.917 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.446 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:20.952 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.473 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:20.978 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.474 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:20.945 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.482 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:20.952 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.485 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:20.991 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.510 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:20.981 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:00.521 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:21.027 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:06.892 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.994+0000 7f0baefe2640 10 monclient: get_monmap_and_config 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.994+0000 7f0baefe2640 10 monclient: build_initial_monmap 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.994+0000 7f0baefe2640 10 monclient: monmap: 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:48:06.995431+0000 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:48:06.995431+0000 2024-03-25T19:48:06.994 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:06.995 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:06.995 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:06.995 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:48:06.995 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:48:06.995 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.995+0000 7f0baefe2640 1 Processor -- start 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.995+0000 7f0baefe2640 1 -- start start 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.995+0000 7f0baefe2640 10 monclient: init 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba80b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba80b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80af940 con 0x7f0ba814be90 2024-03-25T19:48:06.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80b0a70 con 0x7f0ba80b4d60 2024-03-25T19:48:06.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80b0d50 con 0x7f0ba80b1730 2024-03-25T19:48:06.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:06.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:06.996+0000 7f0baefe2640 10 monclient(hunting): authenticate will time out at 5262.259811s 2024-03-25T19:48:07.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.013+0000 7f0bacd7e640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0ba80b4d60 0x7f0ba813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:59040/0 (socket says 172.21.15.50:59040) 2024-03-25T19:48:07.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.013+0000 7f0bacd7e640 1 -- 172.21.15.50:0/1903386427 learned_addr learned my addr 172.21.15.50:0/1903386427 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:07.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.023+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222022370 0 0) 0x7f0ba80b0a70 con 0x7f0ba80b4d60 2024-03-25T19:48:07.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.023+0000 7f0ba77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.023+0000 7f0ba77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.023+0000 7f0ba77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.023+0000 7f0ba77fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:48:07.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.023+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b98003520 con 0x7f0ba80b4d60 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237159832 0 0) 0x7f0b98003520 con 0x7f0ba80b4d60 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 >> v1:172.21.15.50:6790/0 conn(0x7f0ba80b1730 legacy=0x7f0ba80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 >> v1:172.21.15.50:6789/0 conn(0x7f0ba814be90 legacy=0x7f0ba814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 1 monclient: found mon.noname-c 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba80b1030 con 0x7f0ba80b4d60 2024-03-25T19:48:07.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0b9c001690 con 0x7f0ba80b4d60 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0ba77fe640 10 monclient: dump: 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:07.046 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:07.047 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:07.047 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:07.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0baefe2640 5 monclient: authenticate success, global_id 4540 2024-03-25T19:48:07.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.045+0000 7f0baefe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b9c0017f0 con 0x7f0ba80b4d60 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0ba77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0ba77fe640 1 -- 172.21.15.50:0/1903386427 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0b9c001c00 con 0x7f0ba80b4d60 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0ba77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0ba77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0ba77fe640 10 monclient: dump: 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:07.057 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0baefe2640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:07.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.057+0000 7f0baefe2640 4 set_mon_vals no callback set 2024-03-25T19:48:07.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.058+0000 7f0baefe2640 10 monclient: shutdown 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.058+0000 7f0baefe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.058+0000 7f0baefe2640 1 -- 172.21.15.50:0/1903386427 >> v1:172.21.15.76:6789/0 conn(0x7f0ba80b4d60 legacy=0x7f0ba813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.058+0000 7f0baefe2640 1 -- 172.21.15.50:0/1903386427 shutdown_connections 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.058+0000 7f0baefe2640 1 -- 172.21.15.50:0/1903386427 >> 172.21.15.50:0/1903386427 conn(0x7f0ba80617a0 msgr2=0x7f0ba8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.058+0000 7f0baefe2640 1 -- 172.21.15.50:0/1903386427 shutdown_connections 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.058+0000 7f0baefe2640 1 -- 172.21.15.50:0/1903386427 wait complete. 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 10 monclient: build_initial_monmap 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 10 monclient: monmap: 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:07.061 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 1 librados: starting msgr at 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 1 librados: starting objecter 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 1 Processor -- start 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 1 -- start start 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 1 librados: setting wanted keys 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 1 librados: calling monclient init 2024-03-25T19:48:07.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.059+0000 7f0baefe2640 10 monclient: init 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba80b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba80b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8141f40 con 0x7f0ba80b4d60 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8142360 con 0x7f0ba80b1730 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba81424f0 con 0x7f0ba814be90 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:07.063 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.060+0000 7f0baefe2640 10 monclient(hunting): authenticate will time out at 5262.324510s 2024-03-25T19:48:07.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.068+0000 7f0bad57f640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f0ba814be90 0x7f0ba81e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:39662/0 (socket says 172.21.15.50:39662) 2024-03-25T19:48:07.068 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.068+0000 7f0bad57f640 1 -- 172.21.15.50:0/1983796967 learned_addr learned my addr 172.21.15.50:0/1983796967 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:07.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.069+0000 7f0ba57fa640 10 client.?.objecter ms_handle_connect 0x7f0ba814be90 2024-03-25T19:48:07.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.069+0000 7f0ba57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.076+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247545489 0 0) 0x7f0ba80af940 con 0x7f0ba814be90 2024-03-25T19:48:07.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.076+0000 7f0ba57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.076+0000 7f0ba57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.076+0000 7f0ba57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.076+0000 7f0ba57fa640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:48:07.076 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.076+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c004eb0 con 0x7f0ba814be90 2024-03-25T19:48:07.080 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.080+0000 7f0ba57fa640 10 client.?.objecter ms_handle_connect 0x7f0ba80b1730 2024-03-25T19:48:07.080 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.080+0000 7f0ba57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.086+0000 7f0ba57fa640 10 client.?.objecter ms_handle_connect 0x7f0ba80b4d60 2024-03-25T19:48:07.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.086+0000 7f0ba57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:07.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.089+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841085612 0 0) 0x7f0ba8142360 con 0x7f0ba80b1730 2024-03-25T19:48:07.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.089+0000 7f0ba57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.089+0000 7f0ba57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.089+0000 7f0ba57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.089+0000 7f0ba57fa640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:48:07.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.089+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba80af940 con 0x7f0ba80b1730 2024-03-25T19:48:07.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.092+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730438291 0 0) 0x7f0b8c004eb0 con 0x7f0ba814be90 2024-03-25T19:48:07.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.092+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba8142360 con 0x7f0ba814be90 2024-03-25T19:48:07.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.092+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0ba0001af0 con 0x7f0ba814be90 2024-03-25T19:48:07.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.092+0000 7f0ba57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.092+0000 7f0ba57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:07.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.092+0000 7f0ba57fa640 10 monclient(hunting): dump: 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:07.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.092+0000 7f0ba57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:07.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.095+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468893649 0 0) 0x7f0ba8141f40 con 0x7f0ba80b4d60 2024-03-25T19:48:07.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.095+0000 7f0ba57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:07.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.095+0000 7f0ba57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:07.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.095+0000 7f0ba57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:07.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.095+0000 7f0ba57fa640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:48:07.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.095+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c004eb0 con 0x7f0ba80b4d60 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2873912176 0 0) 0x7f0ba8142360 con 0x7f0ba814be90 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 >> v1:172.21.15.76:6789/0 conn(0x7f0ba80b1730 legacy=0x7f0ba81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 >> v1:172.21.15.50:6789/0 conn(0x7f0ba80b4d60 legacy=0x7f0ba81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 1 monclient: found mon.c 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8140d00 con 0x7f0ba814be90 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 10 monclient: _finish_auth 0 2024-03-25T19:48:07.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0ba57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 5 monclient: authenticate success, global_id 4538 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 10 monclient: _renew_subs 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba8138150 con 0x7f0ba814be90 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 10 monclient: _renew_subs 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba81d8520 con 0x7f0ba814be90 2024-03-25T19:48:07.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.102+0000 7f0baefe2640 1 librados: init done 2024-03-25T19:48:07.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.103+0000 7f0baefe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.103+0000 7f0baefe2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.103+0000 7f0baefe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:07.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.103+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba8140f50 con 0x7f0ba814be90 2024-03-25T19:48:07.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.429+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba0001fe0 con 0x7f0ba814be90 2024-03-25T19:48:07.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.429+0000 7f0ba57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:07.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.429+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0ba0003b30 con 0x7f0ba814be90 2024-03-25T19:48:07.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.429+0000 7f0ba57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.429+0000 7f0ba57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.429+0000 7f0ba57fa640 10 monclient: dump: 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:07.430 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:07.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.429+0000 7f0ba6ffd640 4 set_mon_vals no callback set 2024-03-25T19:48:07.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.430+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f0ba0003d50 con 0x7f0ba814be90 2024-03-25T19:48:07.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.431+0000 7f0bacd7e640 10 monclient: get_auth_request con 0x7f0b8c06de00 auth_method 0 2024-03-25T19:48:07.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.431+0000 7f0ba57fa640 10 client.4538.objecter ms_handle_connect 0x7f0b8c06de00 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f0ba0054870 con 0x7f0ba814be90 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 10 client.4538.objecter ms_dispatch 0x7f0ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ba0050380 con 0x7f0ba814be90 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:07.437 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.437+0000 7f0ba57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:07.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.696+0000 7f0baefe2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:07.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.697+0000 7f0baefe2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:07.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.697+0000 7f0baefe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:07.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.697+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ba81d7950 con 0x7f0ba814be90 2024-03-25T19:48:07.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.728+0000 7f0ba57fa640 1 -- 172.21.15.50:0/1983796967 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f0ba81d7950 con 0x7f0ba814be90 2024-03-25T19:48:07.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.728+0000 7f0ba57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:07.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.728+0000 7f0ba57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:07.728 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:07.728 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:07.730 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 10 librados: watch_flush enter 2024-03-25T19:48:07.730 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 10 librados: watch_flush exit 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 10 client.4538.objecter successfully canceled tick 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 >> v1:172.21.15.76:6800/2572556811 conn(0x7f0b8c06de00 legacy=0x7f0b8c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 10 monclient: shutdown 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 >> v1:172.21.15.50:6790/0 conn(0x7f0ba814be90 legacy=0x7f0ba81e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 shutdown_connections 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 >> 172.21.15.50:0/1983796967 conn(0x7f0ba80617a0 msgr2=0x7f0ba80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 shutdown_connections 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 1 -- 172.21.15.50:0/1983796967 wait complete. 2024-03-25T19:48:07.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:07.730+0000 7f0baefe2640 1 librados: shutdown 2024-03-25T19:48:07.742 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:14.548 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:14.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.667+0000 7f1ee84c1640 10 monclient: get_monmap_and_config 2024-03-25T19:48:14.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.667+0000 7f1ee84c1640 10 monclient: build_initial_monmap 2024-03-25T19:48:14.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.667+0000 7f1ee84c1640 10 monclient: monmap: 2024-03-25T19:48:14.667 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:14.667 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:14.667 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:48:14.668648+0000 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:48:14.668648+0000 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:14.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.668+0000 7f1ee84c1640 1 Processor -- start 2024-03-25T19:48:14.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.668+0000 7f1ee84c1640 1 -- start start 2024-03-25T19:48:14.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.668+0000 7f1ee84c1640 10 monclient: init 2024-03-25T19:48:14.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 10 monclient(hunting): picked mon.noname-c con 0x7f1ee00a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 10 monclient(hunting): picked mon.noname-a con 0x7f1ee013f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee01430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee0142520 con 0x7f1ee013f710 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee00b28a0 con 0x7f1ee00a98f0 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee00b2a30 con 0x7f1ee01430c0 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:14.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.669+0000 7f1ee84c1640 10 monclient(hunting): authenticate will time out at 5269.933638s 2024-03-25T19:48:14.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.683+0000 7f1ee625d640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1ee00a98f0 0x7f1ee00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:46364/0 (socket says 172.21.15.50:46364) 2024-03-25T19:48:14.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.683+0000 7f1ee625d640 1 -- 172.21.15.50:0/3501505521 learned_addr learned my addr 172.21.15.50:0/3501505521 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:14.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.694+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006918202 0 0) 0x7f1ee00b28a0 con 0x7f1ee00a98f0 2024-03-25T19:48:14.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.694+0000 7f1ee525b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.694+0000 7f1ee525b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.694+0000 7f1ee525b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.694+0000 7f1ee525b640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:48:14.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.694+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ecc003520 con 0x7f1ee00a98f0 2024-03-25T19:48:14.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.699+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182954111 0 0) 0x7f1ee00b2a30 con 0x7f1ee01430c0 2024-03-25T19:48:14.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.699+0000 7f1ee525b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.699+0000 7f1ee525b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.699+0000 7f1ee525b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.699+0000 7f1ee525b640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:48:14.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.699+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee00b28a0 con 0x7f1ee01430c0 2024-03-25T19:48:14.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.706+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957878130 0 0) 0x7f1ee0142520 con 0x7f1ee013f710 2024-03-25T19:48:14.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.706+0000 7f1ee525b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.706+0000 7f1ee525b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.706+0000 7f1ee525b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.706+0000 7f1ee525b640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:48:14.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.706+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee00b2a30 con 0x7f1ee013f710 2024-03-25T19:48:14.712 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1915249942 0 0) 0x7f1ecc003520 con 0x7f1ee00a98f0 2024-03-25T19:48:14.712 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 >> v1:172.21.15.50:6790/0 conn(0x7f1ee01430c0 legacy=0x7f1ee014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.712 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 >> v1:172.21.15.50:6789/0 conn(0x7f1ee013f710 legacy=0x7f1ee01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 10 monclient: _finish_hunting 0 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 1 monclient: found mon.noname-c 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee00b44c0 con 0x7f1ee00a98f0 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 10 monclient: _finish_auth 0 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 10 monclient: _check_auth_tickets 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1edc001690 con 0x7f1ee00a98f0 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee84c1640 5 monclient: authenticate success, global_id 4546 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee84c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee525b640 10 monclient: dump: 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:14.713 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:14.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.712+0000 7f1ee84c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.724+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1edc0017f0 con 0x7f1ee00a98f0 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.724+0000 7f1ee525b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.724+0000 7f1ee525b640 1 -- 172.21.15.50:0/3501505521 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1edc001c00 con 0x7f1ee00a98f0 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.724+0000 7f1ee84c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.724+0000 7f1ee84c1640 4 set_mon_vals no callback set 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.724+0000 7f1ee525b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.725+0000 7f1ee525b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.725+0000 7f1ee525b640 10 monclient: dump: 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:14.727 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.725+0000 7f1ee84c1640 10 monclient: shutdown 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.725+0000 7f1ee84c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.725+0000 7f1ee84c1640 1 -- 172.21.15.50:0/3501505521 >> v1:172.21.15.76:6789/0 conn(0x7f1ee00a98f0 legacy=0x7f1ee00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.726+0000 7f1ee84c1640 1 -- 172.21.15.50:0/3501505521 shutdown_connections 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.726+0000 7f1ee84c1640 1 -- 172.21.15.50:0/3501505521 >> 172.21.15.50:0/3501505521 conn(0x7f1ee00b7ea0 msgr2=0x7f1ee00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.726+0000 7f1ee84c1640 1 -- 172.21.15.50:0/3501505521 shutdown_connections 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.726+0000 7f1ee84c1640 1 -- 172.21.15.50:0/3501505521 wait complete. 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 10 monclient: build_initial_monmap 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 10 monclient: monmap: 2024-03-25T19:48:14.728 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 1 librados: starting msgr at 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 1 librados: starting objecter 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 1 Processor -- start 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 1 -- start start 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 1 librados: setting wanted keys 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 1 librados: calling monclient init 2024-03-25T19:48:14.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.727+0000 7f1ee84c1640 10 monclient: init 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 10 monclient(hunting): picked mon.noname-a con 0x7f1ee00a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee013f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 10 monclient(hunting): picked mon.noname-c con 0x7f1ee01430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee01d7380 con 0x7f1ee00a98f0 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee0139840 con 0x7f1ee013f710 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee01d8200 con 0x7f1ee01430c0 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:14.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.728+0000 7f1ee84c1640 10 monclient(hunting): authenticate will time out at 5269.992521s 2024-03-25T19:48:14.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.744+0000 7f1ee625d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f1ee00a98f0 0x7f1ee01d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:59552/0 (socket says 172.21.15.50:59552) 2024-03-25T19:48:14.743 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.744+0000 7f1ee625d640 1 -- 172.21.15.50:0/2353806833 learned_addr learned my addr 172.21.15.50:0/2353806833 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:14.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.745+0000 7f1ed6ffd640 10 client.?.objecter ms_handle_connect 0x7f1ee00a98f0 2024-03-25T19:48:14.744 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.745+0000 7f1ed6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:14.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.746+0000 7f1ed6ffd640 10 client.?.objecter ms_handle_connect 0x7f1ee013f710 2024-03-25T19:48:14.745 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.746+0000 7f1ed6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:14.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.753+0000 7f1ed6ffd640 10 client.?.objecter ms_handle_connect 0x7f1ee01430c0 2024-03-25T19:48:14.752 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.753+0000 7f1ed6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:14.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.755+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266123295 0 0) 0x7f1ee0139840 con 0x7f1ee013f710 2024-03-25T19:48:14.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.755+0000 7f1ed6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.755+0000 7f1ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.755+0000 7f1ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.755+0000 7f1ed6ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:48:14.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.755+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec4004eb0 con 0x7f1ee013f710 2024-03-25T19:48:14.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.769+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901546364 0 0) 0x7f1ee01d7380 con 0x7f1ee00a98f0 2024-03-25T19:48:14.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.769+0000 7f1ed6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.769+0000 7f1ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.769+0000 7f1ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.769+0000 7f1ed6ffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:48:14.769 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.769+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee0139840 con 0x7f1ee00a98f0 2024-03-25T19:48:14.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.775+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589217610 0 0) 0x7f1ee01d8200 con 0x7f1ee01430c0 2024-03-25T19:48:14.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.775+0000 7f1ed6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:14.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.775+0000 7f1ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:14.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.775+0000 7f1ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:14.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.775+0000 7f1ed6ffd640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:48:14.775 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.775+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee01d7380 con 0x7f1ee01430c0 2024-03-25T19:48:14.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.780+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3611306222 0 0) 0x7f1ec4004eb0 con 0x7f1ee013f710 2024-03-25T19:48:14.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.780+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ee01d8200 con 0x7f1ee013f710 2024-03-25T19:48:14.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.780+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1ed00018f0 con 0x7f1ee013f710 2024-03-25T19:48:14.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.780+0000 7f1ed6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:14.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.780+0000 7f1ed6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:14.780 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.780+0000 7f1ed6ffd640 10 monclient(hunting): dump: 2024-03-25T19:48:14.780 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:14.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.780+0000 7f1ed6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 664055163 0 0) 0x7f1ee01d8200 con 0x7f1ee013f710 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 >> v1:172.21.15.50:6790/0 conn(0x7f1ee01430c0 legacy=0x7f1ee01daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 >> v1:172.21.15.50:6789/0 conn(0x7f1ee00a98f0 legacy=0x7f1ee01d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 1 monclient: found mon.b 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee01d75d0 con 0x7f1ee013f710 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:14.798 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ed6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 5 monclient: authenticate success, global_id 4549 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 10 monclient: _renew_subs 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee01db6a0 con 0x7f1ee013f710 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 10 monclient: _renew_subs 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee0137910 con 0x7f1ee013f710 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.797+0000 7f1ee84c1640 1 librados: init done 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.798+0000 7f1ee84c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.798+0000 7f1ee84c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.798+0000 7f1ee84c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:14.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:14.798+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee00627b0 con 0x7f1ee013f710 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.102+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ed0003610 con 0x7f1ee013f710 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.102+0000 7f1ed6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.102+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1ed00039a0 con 0x7f1ee013f710 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.102+0000 7f1ed6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.102+0000 7f1ee4a5a640 4 set_mon_vals no callback set 2024-03-25T19:48:15.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.102+0000 7f1ed6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.102+0000 7f1ed6ffd640 10 monclient: dump: 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:15.103 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:15.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.103+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f1ed0003bc0 con 0x7f1ee013f710 2024-03-25T19:48:15.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.104+0000 7f1ee625d640 10 monclient: get_auth_request con 0x7f1ec406ddf0 auth_method 0 2024-03-25T19:48:15.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.104+0000 7f1ed6ffd640 10 client.4549.objecter ms_handle_connect 0x7f1ec406ddf0 2024-03-25T19:48:15.107 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f1ed00535f0 con 0x7f1ee013f710 2024-03-25T19:48:15.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 10 client.4549.objecter ms_dispatch 0x7f1ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:15.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:15.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:15.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T19:48:15.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ed004f100 con 0x7f1ee013f710 2024-03-25T19:48:15.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:15.108 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.108+0000 7f1ed6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:15.377 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.377+0000 7f1ee84c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.377 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.377+0000 7f1ee84c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.377 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.377+0000 7f1ee84c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:15.377 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.377+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ee01d7760 con 0x7f1ee013f710 2024-03-25T19:48:15.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.400+0000 7f1ed6ffd640 1 -- 172.21.15.50:0/2353806833 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f1ed0051030 con 0x7f1ee013f710 2024-03-25T19:48:15.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.400+0000 7f1ed6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:15.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.400+0000 7f1ed6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:15.400 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:15.400 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:15.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 10 librados: watch_flush enter 2024-03-25T19:48:15.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 10 librados: watch_flush exit 2024-03-25T19:48:15.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T19:48:15.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 10 client.4549.objecter successfully canceled tick 2024-03-25T19:48:15.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 >> v1:172.21.15.76:6800/2572556811 conn(0x7f1ec406ddf0 legacy=0x7f1ec40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 10 monclient: shutdown 2024-03-25T19:48:15.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:15.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 >> v1:172.21.15.76:6789/0 conn(0x7f1ee013f710 legacy=0x7f1ee01da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:15.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 shutdown_connections 2024-03-25T19:48:15.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 >> 172.21.15.50:0/2353806833 conn(0x7f1ee00b7ea0 msgr2=0x7f1ee0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:15.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 shutdown_connections 2024-03-25T19:48:15.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 1 -- 172.21.15.50:0/2353806833 wait complete. 2024-03-25T19:48:15.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:15.402+0000 7f1ee84c1640 1 librados: shutdown 2024-03-25T19:48:15.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:20.496 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:41.002 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.515 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:41.021 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.538 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:07:41.044 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.556 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:41.026 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.561 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:41.032 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.577 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:41.083 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.586 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:41.092 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.605 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:07:41.075 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:20.632 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:07:41.138 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.221 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:22.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.339+0000 7fcf0964a640 10 monclient: get_monmap_and_config 2024-03-25T19:48:22.340 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.339+0000 7fcf0964a640 10 monclient: build_initial_monmap 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.339+0000 7fcf0964a640 10 monclient: monmap: 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:48:22.341736+0000 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:48:22.341736+0000 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:48:22.341 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:22.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.340+0000 7fcf0964a640 1 Processor -- start 2024-03-25T19:48:22.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.340+0000 7fcf0964a640 1 -- start start 2024-03-25T19:48:22.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.340+0000 7fcf0964a640 10 monclient: init 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.341+0000 7fcf0964a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.341+0000 7fcf0964a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.341+0000 7fcf0964a640 10 monclient(hunting): picked mon.noname-c con 0x7fcf040a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.341+0000 7fcf0964a640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.341+0000 7fcf0964a640 10 monclient(hunting): picked mon.noname-b con 0x7fcf041430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.341+0000 7fcf0964a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf04142520 con 0x7fcf0413f710 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.341+0000 7fcf0964a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf040b28a0 con 0x7fcf040a98f0 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.342+0000 7fcf0964a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf040b2a30 con 0x7fcf041430c0 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.342+0000 7fcf0964a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:22.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.342+0000 7fcf0964a640 10 monclient(hunting): authenticate will time out at 5277.606743s 2024-03-25T19:48:22.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.354+0000 7fcf027fc640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fcf0413f710 0x7fcf041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:49724/0 (socket says 172.21.15.50:49724) 2024-03-25T19:48:22.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.354+0000 7fcf027fc640 1 -- 172.21.15.50:0/3446522083 learned_addr learned my addr 172.21.15.50:0/3446522083 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:22.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656877060 0 0) 0x7fcf040b28a0 con 0x7fcf040a98f0 2024-03-25T19:48:22.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:48:22.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec003520 con 0x7fcf040a98f0 2024-03-25T19:48:22.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3288989703 0 0) 0x7fcf04142520 con 0x7fcf0413f710 2024-03-25T19:48:22.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:48:22.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.371+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf040b28a0 con 0x7fcf0413f710 2024-03-25T19:48:22.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2927484345 0 0) 0x7fcf040b28a0 con 0x7fcf0413f710 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 >> v1:172.21.15.50:6790/0 conn(0x7fcf041430c0 legacy=0x7fcf0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 >> v1:172.21.15.76:6789/0 conn(0x7fcf040a98f0 legacy=0x7fcf040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 1 monclient: found mon.noname-a 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf040b44c0 con 0x7fcf0413f710 2024-03-25T19:48:22.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcef8001670 con 0x7fcf0413f710 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf0964a640 5 monclient: authenticate success, global_id 4572 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf0964a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf01ffb640 10 monclient: dump: 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:22.385 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:22.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.382+0000 7fcf0964a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef80017d0 con 0x7fcf0413f710 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf01ffb640 1 -- 172.21.15.50:0/3446522083 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcef8001ba0 con 0x7fcf0413f710 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf01ffb640 10 monclient: dump: 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.399 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf0964a640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:22.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.398+0000 7fcf0964a640 4 set_mon_vals no callback set 2024-03-25T19:48:22.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.399+0000 7fcf0964a640 10 monclient: shutdown 2024-03-25T19:48:22.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.399+0000 7fcf0964a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:22.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.399+0000 7fcf0964a640 1 -- 172.21.15.50:0/3446522083 >> v1:172.21.15.50:6789/0 conn(0x7fcf0413f710 legacy=0x7fcf041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.399+0000 7fcf0964a640 1 -- 172.21.15.50:0/3446522083 shutdown_connections 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.399+0000 7fcf0964a640 1 -- 172.21.15.50:0/3446522083 >> 172.21.15.50:0/3446522083 conn(0x7fcf040b7ea0 msgr2=0x7fcf040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.399+0000 7fcf0964a640 1 -- 172.21.15.50:0/3446522083 shutdown_connections 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.399+0000 7fcf0964a640 1 -- 172.21.15.50:0/3446522083 wait complete. 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.400+0000 7fcf0964a640 10 monclient: build_initial_monmap 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.400+0000 7fcf0964a640 10 monclient: monmap: 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:48:22.402 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.400+0000 7fcf0964a640 1 librados: starting msgr at 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.400+0000 7fcf0964a640 1 librados: starting objecter 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 1 Processor -- start 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 1 -- start start 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 1 librados: setting wanted keys 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 1 librados: calling monclient init 2024-03-25T19:48:22.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient: init 2024-03-25T19:48:22.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:22.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:22.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient(hunting): picked mon.noname-c con 0x7fcf040a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:22.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:22.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient(hunting): picked mon.noname-b con 0x7fcf041430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:22.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf041d7830 con 0x7fcf0413f710 2024-03-25T19:48:22.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf041d7500 con 0x7fcf041430c0 2024-03-25T19:48:22.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf04137970 con 0x7fcf040a98f0 2024-03-25T19:48:22.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:22.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.401+0000 7fcf0964a640 10 monclient(hunting): authenticate will time out at 5277.666641s 2024-03-25T19:48:22.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.405+0000 7fcf02ffd640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fcf040a98f0 0x7fcf04138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:58814/0 (socket says 172.21.15.50:58814) 2024-03-25T19:48:22.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.405+0000 7fcf02ffd640 1 -- 172.21.15.50:0/2103471800 learned_addr learned my addr 172.21.15.50:0/2103471800 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:22.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.411+0000 7fceeb7fe640 10 client.?.objecter ms_handle_connect 0x7fcf040a98f0 2024-03-25T19:48:22.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.411+0000 7fceeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.414+0000 7fceeb7fe640 10 client.?.objecter ms_handle_connect 0x7fcf0413f710 2024-03-25T19:48:22.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.414+0000 7fceeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 10 client.?.objecter ms_handle_connect 0x7fcf041430c0 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402607485 0 0) 0x7fcf04137970 con 0x7fcf040a98f0 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:48:22.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.416+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef0004eb0 con 0x7fcf040a98f0 2024-03-25T19:48:22.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.422+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2833494726 0 0) 0x7fcf041d7500 con 0x7fcf041430c0 2024-03-25T19:48:22.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.422+0000 7fceeb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:22.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.422+0000 7fceeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:22.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.422+0000 7fceeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:22.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.422+0000 7fceeb7fe640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:48:22.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.422+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf04137970 con 0x7fcf041430c0 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.433+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2969440231 0 0) 0x7fcef0004eb0 con 0x7fcf040a98f0 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.433+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf041d7500 con 0x7fcf040a98f0 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.433+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcef4001a10 con 0x7fcf040a98f0 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.433+0000 7fceeb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.433+0000 7fceeb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.433+0000 7fceeb7fe640 10 monclient(hunting): dump: 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.434 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.435 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:22.435 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:22.435 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:22.435 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:22.435 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:22.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.433+0000 7fceeb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:22.438 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.437+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015586257 0 0) 0x7fcf04137970 con 0x7fcf041430c0 2024-03-25T19:48:22.438 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.437+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcef0004eb0 con 0x7fcf041430c0 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.438+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcefc001b00 con 0x7fcf041430c0 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.438+0000 7fceeb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.438+0000 7fceeb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.438+0000 7fceeb7fe640 10 monclient(hunting): dump: 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:22.439 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4068175840 0 0) 0x7fcf041d7500 con 0x7fcf040a98f0 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 >> v1:172.21.15.76:6789/0 conn(0x7fcf041430c0 legacy=0x7fcf041e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 >> v1:172.21.15.50:6789/0 conn(0x7fcf0413f710 legacy=0x7fcf041d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 1 monclient: found mon.c 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04137b00 con 0x7fcf040a98f0 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:22.441 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fceeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fcf0964a640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fcf0964a640 10 monclient: _renew_subs 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fcf0964a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf04135080 con 0x7fcf040a98f0 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fcf0964a640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fcf0964a640 10 monclient: _renew_subs 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.439+0000 7fcf0964a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.440+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf041e36d0 con 0x7fcf040a98f0 2024-03-25T19:48:22.442 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.440+0000 7fcf0964a640 1 librados: init done 2024-03-25T19:48:22.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.441+0000 7fcf0964a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.441+0000 7fcf0964a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.441+0000 7fcf0964a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:22.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.441+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf041d7690 con 0x7fcf040a98f0 2024-03-25T19:48:22.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.781+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcef4001c50 con 0x7fcf040a98f0 2024-03-25T19:48:22.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.781+0000 7fceeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:22.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.781+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fcef40038c0 con 0x7fcf040a98f0 2024-03-25T19:48:22.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.781+0000 7fceeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:22.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.781+0000 7fceeb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:22.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.781+0000 7fceeb7fe640 10 monclient: dump: 2024-03-25T19:48:22.785 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:22.785 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:22.785 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.785 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:22.785 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:22.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.781+0000 7fcf017fa640 4 set_mon_vals no callback set 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.782+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fcef4003aa0 con 0x7fcf040a98f0 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.782+0000 7fcf027fc640 10 monclient: get_auth_request con 0x7fcef006df20 auth_method 0 2024-03-25T19:48:22.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.783+0000 7fceeb7fe640 10 client.4550.objecter ms_handle_connect 0x7fcef006df20 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fcef4054700 con 0x7fcf040a98f0 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 10 client.4550.objecter ms_dispatch 0x7fcf04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcef4050210 con 0x7fcf040a98f0 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:22.791 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:22.789+0000 7fceeb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:23.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.016+0000 7fcf0964a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.016+0000 7fcf0964a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.016+0000 7fcf0964a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:48:23.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.016+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf04137c90 con 0x7fcf040a98f0 2024-03-25T19:48:23.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.046+0000 7fceeb7fe640 1 -- 172.21.15.50:0/2103471800 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fcf04137c90 con 0x7fcf040a98f0 2024-03-25T19:48:23.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.046+0000 7fceeb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:23.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.046+0000 7fceeb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:23.047 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:23.047 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:23.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 10 librados: watch_flush enter 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 10 librados: watch_flush exit 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 10 client.4550.objecter successfully canceled tick 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 >> v1:172.21.15.76:6800/2572556811 conn(0x7fcef006df20 legacy=0x7fcef00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 10 monclient: shutdown 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.048+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 >> v1:172.21.15.50:6790/0 conn(0x7fcf040a98f0 legacy=0x7fcf04138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.049+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 shutdown_connections 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.049+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 >> 172.21.15.50:0/2103471800 conn(0x7fcf040b7ea0 msgr2=0x7fcf04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.049+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 shutdown_connections 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.049+0000 7fcf0964a640 1 -- 172.21.15.50:0/2103471800 wait complete. 2024-03-25T19:48:23.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:23.049+0000 7fcf0964a640 1 librados: shutdown 2024-03-25T19:48:23.061 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:29.868 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.980+0000 7feab22c3640 10 monclient: get_monmap_and_config 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.980+0000 7feab22c3640 10 monclient: build_initial_monmap 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.980+0000 7feab22c3640 10 monclient: monmap: 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:48:29.982323+0000 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:48:29.982323+0000 2024-03-25T19:48:29.981 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:29.982 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:29.982 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:29.982 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:48:29.982 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:48:29.982 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.981+0000 7feab22c3640 1 Processor -- start 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.981+0000 7feab22c3640 1 -- start start 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.981+0000 7feab22c3640 10 monclient: init 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 10 monclient(hunting): picked mon.noname-c con 0x7feaac0a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 10 monclient(hunting): picked mon.noname-a con 0x7feaac13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 10 monclient(hunting): picked mon.noname-b con 0x7feaac1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac142520 con 0x7feaac13f710 2024-03-25T19:48:29.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0b28a0 con 0x7feaac0a98f0 2024-03-25T19:48:29.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0b2a30 con 0x7feaac1430c0 2024-03-25T19:48:29.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:29.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.982+0000 7feab22c3640 10 monclient(hunting): authenticate will time out at 5285.247156s 2024-03-25T19:48:29.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.985+0000 7feab0860640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7feaac1430c0 0x7feaac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:60840/0 (socket says 172.21.15.50:60840) 2024-03-25T19:48:29.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:29.985+0000 7feab0860640 1 -- 172.21.15.50:0/155223149 learned_addr learned my addr 172.21.15.50:0/155223149 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:30.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.001+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025641027 0 0) 0x7feaac0b28a0 con 0x7feaac0a98f0 2024-03-25T19:48:30.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.002+0000 7feaaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.002+0000 7feaaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.002+0000 7feaaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.002+0000 7feaaaffd640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:48:30.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.002+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea98003520 con 0x7feaac0a98f0 2024-03-25T19:48:30.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.011+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627554564 0 0) 0x7feaac0b2a30 con 0x7feaac1430c0 2024-03-25T19:48:30.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.011+0000 7feaaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.011+0000 7feaaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.011+0000 7feaaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.011+0000 7feaaaffd640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:48:30.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.011+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaac0b28a0 con 0x7feaac1430c0 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 500586974 0 0) 0x7fea98003520 con 0x7feaac0a98f0 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 >> v1:172.21.15.50:6790/0 conn(0x7feaac1430c0 legacy=0x7feaac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 >> v1:172.21.15.50:6789/0 conn(0x7feaac13f710 legacy=0x7feaac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 1 monclient: found mon.noname-c 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:48:30.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac0b44c0 con 0x7feaac0a98f0 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feaa4001a10 con 0x7feaac0a98f0 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feab22c3640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feab22c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feaaaffd640 10 monclient: dump: 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.025 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.023+0000 7feab22c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feaaaffd640 1 -- 172.21.15.50:0/155223149 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaa4001b70 con 0x7feaac0a98f0 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feaaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:30.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:30.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 4 set_mon_vals no callback set 2024-03-25T19:48:30.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 10 monclient: shutdown 2024-03-25T19:48:30.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:30.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 1 -- 172.21.15.50:0/155223149 >> v1:172.21.15.76:6789/0 conn(0x7feaac0a98f0 legacy=0x7feaac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 1 -- 172.21.15.50:0/155223149 shutdown_connections 2024-03-25T19:48:30.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 1 -- 172.21.15.50:0/155223149 >> 172.21.15.50:0/155223149 conn(0x7feaac0b7ea0 msgr2=0x7feaac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:30.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.026+0000 7feab22c3640 1 -- 172.21.15.50:0/155223149 shutdown_connections 2024-03-25T19:48:30.028 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.027+0000 7feab22c3640 1 -- 172.21.15.50:0/155223149 wait complete. 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.027+0000 7feab22c3640 10 monclient: build_initial_monmap 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.027+0000 7feab22c3640 10 monclient: monmap: 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:30.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.028+0000 7feab22c3640 1 librados: starting msgr at 2024-03-25T19:48:30.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.028+0000 7feab22c3640 1 librados: starting objecter 2024-03-25T19:48:30.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.028+0000 7feab22c3640 1 Processor -- start 2024-03-25T19:48:30.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.028+0000 7feab22c3640 1 -- start start 2024-03-25T19:48:30.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.028+0000 7feab22c3640 1 librados: setting wanted keys 2024-03-25T19:48:30.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.028+0000 7feab22c3640 1 librados: calling monclient init 2024-03-25T19:48:30.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.028+0000 7feab22c3640 10 monclient: init 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 10 monclient(hunting): picked mon.noname-b con 0x7feaac0a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 10 monclient(hunting): picked mon.noname-a con 0x7feaac13f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 10 monclient(hunting): picked mon.noname-c con 0x7feaac1430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0aba50 con 0x7feaac13f710 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0abd30 con 0x7feaac0a98f0 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaac0ac010 con 0x7feaac1430c0 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:30.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.029+0000 7feab22c3640 10 monclient(hunting): authenticate will time out at 5285.294250s 2024-03-25T19:48:30.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.035+0000 7feaab7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7feaac13f710 0x7feaac0ac430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54650/0 (socket says 172.21.15.50:54650) 2024-03-25T19:48:30.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.035+0000 7feaab7fe640 1 -- 172.21.15.50:0/4200949234 learned_addr learned my addr 172.21.15.50:0/4200949234 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:30.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.044+0000 7feaa8ff9640 10 client.?.objecter ms_handle_connect 0x7feaac13f710 2024-03-25T19:48:30.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.044+0000 7feaa8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.048+0000 7feaa8ff9640 10 client.?.objecter ms_handle_connect 0x7feaac0a98f0 2024-03-25T19:48:30.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.048+0000 7feaa8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.052+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546621739 0 0) 0x7feaac0aba50 con 0x7feaac13f710 2024-03-25T19:48:30.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.052+0000 7feaa8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.052+0000 7feaa8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.052+0000 7feaa8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.052+0000 7feaa8ff9640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:48:30.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.052+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea8c004eb0 con 0x7feaac13f710 2024-03-25T19:48:30.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.057+0000 7feaa8ff9640 10 client.?.objecter ms_handle_connect 0x7feaac1430c0 2024-03-25T19:48:30.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.057+0000 7feaa8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:30.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.069+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209127183 0 0) 0x7feaac0ac010 con 0x7feaac1430c0 2024-03-25T19:48:30.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.069+0000 7feaa8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.069+0000 7feaa8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.069+0000 7feaa8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.069+0000 7feaa8ff9640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:48:30.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.069+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaac0aba50 con 0x7feaac1430c0 2024-03-25T19:48:30.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.070+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471520393 0 0) 0x7feaac0abd30 con 0x7feaac0a98f0 2024-03-25T19:48:30.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.070+0000 7feaa8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:30.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.070+0000 7feaa8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:30.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.070+0000 7feaa8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:30.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.070+0000 7feaa8ff9640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:48:30.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.070+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaac0ac010 con 0x7feaac0a98f0 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505978251 0 0) 0x7feaac0ac010 con 0x7feaac0a98f0 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaac0abd30 con 0x7feaac0a98f0 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feaa4002020 con 0x7feaac0a98f0 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.082 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667242300 0 0) 0x7fea8c004eb0 con 0x7feaac13f710 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaac0ac010 con 0x7feaac13f710 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fea9c001a30 con 0x7feaac13f710 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.081+0000 7feaa8ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:30.083 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.084 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.084 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.084 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:30.084 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:30.084 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:30.084 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:30.084 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.101+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3971154599 0 0) 0x7feaac0aba50 con 0x7feaac1430c0 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.101+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea8c004eb0 con 0x7feaac1430c0 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.101+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fea94001810 con 0x7feaac1430c0 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.101+0000 7feaa8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.101+0000 7feaa8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.101+0000 7feaa8ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.102 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.103 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:30.103 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:30.103 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:30.103 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:30.103 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:30.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3796928403 0 0) 0x7feaac0ac010 con 0x7feaac13f710 2024-03-25T19:48:30.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 >> v1:172.21.15.50:6790/0 conn(0x7feaac1430c0 legacy=0x7feaac1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 >> v1:172.21.15.76:6789/0 conn(0x7feaac0a98f0 legacy=0x7feaac1363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:48:30.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 1 monclient: found mon.a 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac138fe0 con 0x7feaac13f710 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feaa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feab22c3640 5 monclient: authenticate success, global_id 4581 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feab22c3640 10 monclient: _renew_subs 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feab22c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.102+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac137cf0 con 0x7feaac13f710 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.103+0000 7feab22c3640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.103+0000 7feab22c3640 10 monclient: _renew_subs 2024-03-25T19:48:30.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.103+0000 7feab22c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:30.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.103+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac138ac0 con 0x7feaac13f710 2024-03-25T19:48:30.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.103+0000 7feab22c3640 1 librados: init done 2024-03-25T19:48:30.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.104+0000 7feab22c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.104+0000 7feab22c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.104+0000 7feab22c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:30.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.104+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac0627b0 con 0x7feaac13f710 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.456+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea9c001c70 con 0x7feaac13f710 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.456+0000 7feaa8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.456+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fea9c003a60 con 0x7feaac13f710 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.456+0000 7feaa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.456+0000 7feaa8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.456+0000 7feaa8ff9640 10 monclient: dump: 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.456 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.456+0000 7feaaa7fc640 4 set_mon_vals no callback set 2024-03-25T19:48:30.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.457+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fea9c003c40 con 0x7feaac13f710 2024-03-25T19:48:30.458 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.458+0000 7feaabfff640 10 monclient: get_auth_request con 0x7fea8c06e110 auth_method 0 2024-03-25T19:48:30.458 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.458+0000 7feaa8ff9640 10 client.4581.objecter ms_handle_connect 0x7fea8c06e110 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fea9c054890 con 0x7feaac13f710 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 10 client.4581.objecter ms_dispatch 0x7feaac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T19:48:30.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea9c0503a0 con 0x7feaac13f710 2024-03-25T19:48:30.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:30.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.466+0000 7feaa8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:30.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.719+0000 7feab22c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:30.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.719+0000 7feab22c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:30.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.719+0000 7feab22c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:30.719 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.719+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaac139170 con 0x7feaac13f710 2024-03-25T19:48:30.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.773+0000 7feaa8ff9640 1 -- 172.21.15.50:0/4200949234 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fea9c0522d0 con 0x7feaac13f710 2024-03-25T19:48:30.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.773+0000 7feaa8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:30.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.773+0000 7feaa8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:30.773 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:30.773 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.775+0000 7feab22c3640 10 librados: watch_flush enter 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.775+0000 7feab22c3640 10 librados: watch_flush exit 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.775+0000 7feab22c3640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.775+0000 7feab22c3640 10 client.4581.objecter successfully canceled tick 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 >> v1:172.21.15.76:6800/2572556811 conn(0x7fea8c06e110 legacy=0x7fea8c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 10 monclient: shutdown 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 >> v1:172.21.15.50:6789/0 conn(0x7feaac13f710 legacy=0x7feaac0ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:30.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 shutdown_connections 2024-03-25T19:48:30.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 >> 172.21.15.50:0/4200949234 conn(0x7feaac0b7ea0 msgr2=0x7feaac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:30.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 shutdown_connections 2024-03-25T19:48:30.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 1 -- 172.21.15.50:0/4200949234 wait complete. 2024-03-25T19:48:30.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:30.776+0000 7feab22c3640 1 librados: shutdown 2024-03-25T19:48:30.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:37.595 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.783+0000 7f122c125640 10 monclient: get_monmap_and_config 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.783+0000 7f122c125640 10 monclient: build_initial_monmap 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.783+0000 7f122c125640 10 monclient: monmap: 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:48:37.784248+0000 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:48:37.784248+0000 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:48:37.783 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:48:37.784 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.784+0000 7f122c125640 1 Processor -- start 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.784+0000 7f122c125640 1 -- start start 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.784+0000 7f122c125640 10 monclient: init 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 10 monclient(hunting): picked mon.noname-a con 0x7f12240b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 10 monclient(hunting): picked mon.noname-b con 0x7f12240b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 10 monclient(hunting): picked mon.noname-c con 0x7f122414be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12240af940 con 0x7f12240b4d60 2024-03-25T19:48:37.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12240b0a70 con 0x7f122414be90 2024-03-25T19:48:37.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12240b0d50 con 0x7f12240b1730 2024-03-25T19:48:37.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:37.786 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.785+0000 7f122c125640 10 monclient(hunting): authenticate will time out at 5293.049228s 2024-03-25T19:48:37.795 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.796+0000 7f12296c0640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f12240b1730 0x7f12240b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:59408/0 (socket says 172.21.15.50:59408) 2024-03-25T19:48:37.796 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.796+0000 7f12296c0640 1 -- 172.21.15.50:0/3246438045 learned_addr learned my addr 172.21.15.50:0/3246438045 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:37.806 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.807+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593321799 0 0) 0x7f12240b0d50 con 0x7f12240b1730 2024-03-25T19:48:37.806 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.807+0000 7f1228ebf640 10 monclient(hunting): _negotiate 2024-03-25T19:48:37.806 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.807+0000 7f1228ebf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:37.807 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.807+0000 7f1228ebf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:37.807 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.807+0000 7f1228ebf640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:48:37.807 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.807+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1210003520 con 0x7f12240b1730 2024-03-25T19:48:37.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.812+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216573007 0 0) 0x7f12240b0a70 con 0x7f122414be90 2024-03-25T19:48:37.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.812+0000 7f1228ebf640 10 monclient(hunting): _negotiate 2024-03-25T19:48:37.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.812+0000 7f1228ebf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:37.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.812+0000 7f1228ebf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:37.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.812+0000 7f1228ebf640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:48:37.812 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.812+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12240b0d50 con 0x7f122414be90 2024-03-25T19:48:37.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878176547 0 0) 0x7f1210003520 con 0x7f12240b1730 2024-03-25T19:48:37.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 >> v1:172.21.15.76:6789/0 conn(0x7f122414be90 legacy=0x7f122414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 >> v1:172.21.15.50:6789/0 conn(0x7f12240b4d60 legacy=0x7f122413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 10 monclient: _finish_hunting 0 2024-03-25T19:48:37.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 1 monclient: found mon.noname-b 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12240b1030 con 0x7f12240b1730 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 10 monclient: _finish_auth 0 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 10 monclient: _check_auth_tickets 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1214001690 con 0x7f12240b1730 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f122c125640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f122c125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f1228ebf640 10 monclient: dump: 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:37.819 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:37.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.818+0000 7f122c125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12140017f0 con 0x7f12240b1730 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f1228ebf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f1228ebf640 1 -- 172.21.15.50:0/3246438045 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1214001c00 con 0x7f12240b1730 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f1228ebf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f1228ebf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f1228ebf640 10 monclient: dump: 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:37.835 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f122c125640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:37.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.835+0000 7f122c125640 4 set_mon_vals no callback set 2024-03-25T19:48:37.837 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.836+0000 7f122c125640 10 monclient: shutdown 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.836+0000 7f122c125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.836+0000 7f122c125640 1 -- 172.21.15.50:0/3246438045 >> v1:172.21.15.50:6790/0 conn(0x7f12240b1730 legacy=0x7f12240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.836+0000 7f122c125640 1 -- 172.21.15.50:0/3246438045 shutdown_connections 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.836+0000 7f122c125640 1 -- 172.21.15.50:0/3246438045 >> 172.21.15.50:0/3246438045 conn(0x7f12240617a0 msgr2=0x7f1224061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.836+0000 7f122c125640 1 -- 172.21.15.50:0/3246438045 shutdown_connections 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.836+0000 7f122c125640 1 -- 172.21.15.50:0/3246438045 wait complete. 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.837+0000 7f122c125640 10 monclient: build_initial_monmap 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.837+0000 7f122c125640 10 monclient: monmap: 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:48:37.838 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.837+0000 7f122c125640 1 librados: starting msgr at 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.837+0000 7f122c125640 1 librados: starting objecter 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.837+0000 7f122c125640 1 Processor -- start 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 1 -- start start 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 1 librados: setting wanted keys 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 1 librados: calling monclient init 2024-03-25T19:48:37.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 10 monclient: init 2024-03-25T19:48:37.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 10 monclient(hunting): picked mon.noname-b con 0x7f12240b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.838+0000 7f122c125640 10 monclient(hunting): picked mon.noname-a con 0x7f12240b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.839+0000 7f122c125640 10 monclient(hunting): picked mon.noname-c con 0x7f122414be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.839+0000 7f122c125640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1224141bf0 con 0x7f12240b4d60 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.839+0000 7f122c125640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1224141ed0 con 0x7f12240b1730 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.839+0000 7f122c125640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12241421b0 con 0x7f122414be90 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.839+0000 7f122c125640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.839+0000 7f122c125640 10 monclient(hunting): authenticate will time out at 5293.102870s 2024-03-25T19:48:37.841 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.842+0000 7f12296c0640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f12240b4d60 0x7f1224133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:40204/0 (socket says 172.21.15.50:40204) 2024-03-25T19:48:37.842 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.842+0000 7f12296c0640 1 -- 172.21.15.50:0/1956733231 learned_addr learned my addr 172.21.15.50:0/1956733231 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:37.842 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.843+0000 7f121a7fc640 10 client.?.objecter ms_handle_connect 0x7f12240b4d60 2024-03-25T19:48:37.842 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.843+0000 7f121a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:37.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.858+0000 7f121a7fc640 10 client.?.objecter ms_handle_connect 0x7f12240b1730 2024-03-25T19:48:37.858 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.858+0000 7f121a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:37.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.859+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875977366 0 0) 0x7f1224141bf0 con 0x7f12240b4d60 2024-03-25T19:48:37.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.859+0000 7f121a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:37.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.859+0000 7f121a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:37.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.859+0000 7f121a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:37.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.859+0000 7f121a7fc640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:48:37.860 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.859+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1208004eb0 con 0x7f12240b4d60 2024-03-25T19:48:37.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.861+0000 7f121a7fc640 10 client.?.objecter ms_handle_connect 0x7f122414be90 2024-03-25T19:48:37.861 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.861+0000 7f121a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:37.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.876+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741610541 0 0) 0x7f1224141ed0 con 0x7f12240b1730 2024-03-25T19:48:37.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.876+0000 7f121a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:37.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.876+0000 7f121a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:37.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.876+0000 7f121a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:37.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.876+0000 7f121a7fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:48:37.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.876+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1224141bf0 con 0x7f12240b1730 2024-03-25T19:48:37.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.879+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118925459 0 0) 0x7f12241421b0 con 0x7f122414be90 2024-03-25T19:48:37.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.879+0000 7f121a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:37.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.879+0000 7f121a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:37.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.879+0000 7f121a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:37.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.879+0000 7f121a7fc640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:48:37.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.879+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1224141ed0 con 0x7f122414be90 2024-03-25T19:48:37.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.880+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2743167005 0 0) 0x7f1208004eb0 con 0x7f12240b4d60 2024-03-25T19:48:37.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.880+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12241421b0 con 0x7f12240b4d60 2024-03-25T19:48:37.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.880+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1214001c40 con 0x7f12240b4d60 2024-03-25T19:48:37.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.880+0000 7f121a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:37.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.880+0000 7f121a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:37.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.880+0000 7f121a7fc640 10 monclient(hunting): dump: 2024-03-25T19:48:37.880 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:37.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.880+0000 7f121a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:37.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.895+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1305770042 0 0) 0x7f1224141ed0 con 0x7f122414be90 2024-03-25T19:48:37.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.895+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1208004eb0 con 0x7f122414be90 2024-03-25T19:48:37.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.896+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f121c001bd0 con 0x7f122414be90 2024-03-25T19:48:37.895 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.896+0000 7f121a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.896+0000 7f121a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.896+0000 7f121a7fc640 10 monclient(hunting): dump: 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:37.896 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:37.898 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 315672636 0 0) 0x7f12241421b0 con 0x7f12240b4d60 2024-03-25T19:48:37.898 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 >> v1:172.21.15.50:6790/0 conn(0x7f122414be90 legacy=0x7f12241341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.898 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 >> v1:172.21.15.76:6789/0 conn(0x7f12240b1730 legacy=0x7f12241425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 1 monclient: found mon.a 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1224140f00 con 0x7f12240b4d60 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f121a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 5 monclient: authenticate success, global_id 4587 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 10 monclient: _renew_subs 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12241406f0 con 0x7f12240b4d60 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 10 monclient: _renew_subs 2024-03-25T19:48:37.899 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:37.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1224140380 con 0x7f12240b4d60 2024-03-25T19:48:37.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.898+0000 7f122c125640 1 librados: init done 2024-03-25T19:48:37.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.899+0000 7f122c125640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:37.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.899+0000 7f122c125640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:37.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.899+0000 7f122c125640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:37.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:37.899+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12241401f0 con 0x7f12240b4d60 2024-03-25T19:48:38.219 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.219+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1214004a90 con 0x7f12240b4d60 2024-03-25T19:48:38.219 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.219+0000 7f121a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:38.219 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.219+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1214004de0 con 0x7f12240b4d60 2024-03-25T19:48:38.219 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.219+0000 7f121a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.219+0000 7f121a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.219+0000 7f121a7fc640 10 monclient: dump: 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:38.220 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.219+0000 7f121bfff640 4 set_mon_vals no callback set 2024-03-25T19:48:38.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.220+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f1214005000 con 0x7f12240b4d60 2024-03-25T19:48:38.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.221+0000 7f1229ec1640 10 monclient: get_auth_request con 0x7f120806df90 auth_method 0 2024-03-25T19:48:38.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.221+0000 7f121a7fc640 10 client.4587.objecter ms_handle_connect 0x7f120806df90 2024-03-25T19:48:38.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f1214054950 con 0x7f12240b4d60 2024-03-25T19:48:38.225 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 10 client.4587.objecter ms_dispatch 0x7f1224000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:38.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:38.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:38.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T19:48:38.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1214050460 con 0x7f12240b4d60 2024-03-25T19:48:38.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:38.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.226+0000 7f121a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:38.479 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.479+0000 7f122c125640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.479 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.479+0000 7f122c125640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.479 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.479+0000 7f122c125640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:38.480 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.479+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12240627b0 con 0x7f12240b4d60 2024-03-25T19:48:38.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.518+0000 7f121a7fc640 1 -- 172.21.15.50:0/1956733231 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f1214052390 con 0x7f12240b4d60 2024-03-25T19:48:38.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.518+0000 7f121a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:38.518 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.518+0000 7f121a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:38.518 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:38.518 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:38.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 10 librados: watch_flush enter 2024-03-25T19:48:38.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 10 librados: watch_flush exit 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 10 client.4587.objecter successfully canceled tick 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 >> v1:172.21.15.76:6800/2572556811 conn(0x7f120806df90 legacy=0x7f1208070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 10 monclient: shutdown 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 >> v1:172.21.15.50:6789/0 conn(0x7f12240b4d60 legacy=0x7f1224133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 shutdown_connections 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 >> 172.21.15.50:0/1956733231 conn(0x7f12240617a0 msgr2=0x7f122413c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 shutdown_connections 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 1 -- 172.21.15.50:0/1956733231 wait complete. 2024-03-25T19:48:38.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:38.521+0000 7f122c125640 1 librados: shutdown 2024-03-25T19:48:38.534 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:40.574 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:01.080 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.581 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:01.087 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.612 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:01.118 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.647 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:01.118 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.655 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:01.161 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.657 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:01.127 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.660 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:01.166 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.691 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:01.196 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:40.691 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:01.162 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.339 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.459+0000 7fca5f8ba640 10 monclient: get_monmap_and_config 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.459+0000 7fca5f8ba640 10 monclient: build_initial_monmap 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.459+0000 7fca5f8ba640 10 monclient: monmap: 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:48:45.460244+0000 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:48:45.460244+0000 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:48:45.459 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:48:45.460 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 1 Processor -- start 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 1 -- start start 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient: init 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient(hunting): picked mon.noname-b con 0x7fca580b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient(hunting): picked mon.noname-a con 0x7fca580b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient(hunting): picked mon.noname-c con 0x7fca5814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca580af940 con 0x7fca580b1730 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca580b0a70 con 0x7fca5814be90 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca580b0d50 con 0x7fca580b4d60 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:45.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.460+0000 7fca5f8ba640 10 monclient(hunting): authenticate will time out at 5300.724564s 2024-03-25T19:48:45.474 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.475+0000 7fca5de57640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fca5814be90 0x7fca5814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:39540/0 (socket says 172.21.15.50:39540) 2024-03-25T19:48:45.474 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.475+0000 7fca5de57640 1 -- 172.21.15.50:0/2377577349 learned_addr learned my addr 172.21.15.50:0/2377577349 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:45.487 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.488+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277392495 0 0) 0x7fca580b0d50 con 0x7fca580b4d60 2024-03-25T19:48:45.487 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.488+0000 7fca4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.488+0000 7fca4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.488+0000 7fca4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.488+0000 7fca4ffff640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:48:45.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.488+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca44003520 con 0x7fca580b4d60 2024-03-25T19:48:45.492 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.493+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312604706 0 0) 0x7fca580b0a70 con 0x7fca5814be90 2024-03-25T19:48:45.492 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.493+0000 7fca4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.493 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.493+0000 7fca4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.493 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.493+0000 7fca4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.493 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.493+0000 7fca4ffff640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:48:45.493 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.493+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca580b0d50 con 0x7fca5814be90 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 140835665 0 0) 0x7fca44003520 con 0x7fca580b4d60 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 >> v1:172.21.15.76:6789/0 conn(0x7fca5814be90 legacy=0x7fca5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 >> v1:172.21.15.50:6789/0 conn(0x7fca580b1730 legacy=0x7fca580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 1 monclient: found mon.noname-b 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca580b1030 con 0x7fca580b4d60 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 10 monclient: _finish_auth 0 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:45.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fca54001690 con 0x7fca580b4d60 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca5f8ba640 5 monclient: authenticate success, global_id 4565 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca5f8ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca4ffff640 10 monclient: dump: 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:45.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.506+0000 7fca5f8ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:45.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca540017f0 con 0x7fca580b4d60 2024-03-25T19:48:45.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:45.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca4ffff640 1 -- 172.21.15.50:0/2377577349 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fca54001c00 con 0x7fca580b4d60 2024-03-25T19:48:45.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca5f8ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:45.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca5f8ba640 4 set_mon_vals no callback set 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.547+0000 7fca4ffff640 10 monclient: dump: 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:45.548 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.548+0000 7fca5f8ba640 10 monclient: shutdown 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.548+0000 7fca5f8ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.548+0000 7fca5f8ba640 1 -- 172.21.15.50:0/2377577349 >> v1:172.21.15.50:6790/0 conn(0x7fca580b4d60 legacy=0x7fca5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.548+0000 7fca5f8ba640 1 -- 172.21.15.50:0/2377577349 shutdown_connections 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.548+0000 7fca5f8ba640 1 -- 172.21.15.50:0/2377577349 >> 172.21.15.50:0/2377577349 conn(0x7fca580617a0 msgr2=0x7fca58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.548+0000 7fca5f8ba640 1 -- 172.21.15.50:0/2377577349 shutdown_connections 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.548+0000 7fca5f8ba640 1 -- 172.21.15.50:0/2377577349 wait complete. 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.549+0000 7fca5f8ba640 10 monclient: build_initial_monmap 2024-03-25T19:48:45.550 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.549+0000 7fca5f8ba640 10 monclient: monmap: 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.549+0000 7fca5f8ba640 1 librados: starting msgr at 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.549+0000 7fca5f8ba640 1 librados: starting objecter 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 1 Processor -- start 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 1 -- start start 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 1 librados: setting wanted keys 2024-03-25T19:48:45.551 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 1 librados: calling monclient init 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 10 monclient: init 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.550+0000 7fca5f8ba640 10 monclient(hunting): picked mon.noname-c con 0x7fca580b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.551+0000 7fca5f8ba640 10 monclient(hunting): picked mon.noname-b con 0x7fca580b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.551+0000 7fca5f8ba640 10 monclient(hunting): picked mon.noname-a con 0x7fca5814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.551+0000 7fca5f8ba640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca58143900 con 0x7fca5814be90 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.551+0000 7fca5f8ba640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5813b550 con 0x7fca580b4d60 2024-03-25T19:48:45.552 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.551+0000 7fca5f8ba640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5813b7a0 con 0x7fca580b1730 2024-03-25T19:48:45.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.551+0000 7fca5f8ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:45.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.551+0000 7fca5f8ba640 10 monclient(hunting): authenticate will time out at 5300.814892s 2024-03-25T19:48:45.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.553+0000 7fca5ce55640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fca580b4d60 0x7fca5812fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:39606/0 (socket says 172.21.15.50:39606) 2024-03-25T19:48:45.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.553+0000 7fca5ce55640 1 -- 172.21.15.50:0/3401241294 learned_addr learned my addr 172.21.15.50:0/3401241294 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:45.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.556+0000 7fca4dffb640 10 client.?.objecter ms_handle_connect 0x7fca580b4d60 2024-03-25T19:48:45.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.556+0000 7fca4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.566+0000 7fca4dffb640 10 client.?.objecter ms_handle_connect 0x7fca5814be90 2024-03-25T19:48:45.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.566+0000 7fca4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.567+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142940451 0 0) 0x7fca5813b550 con 0x7fca580b4d60 2024-03-25T19:48:45.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.567+0000 7fca4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.567+0000 7fca4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.567 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.567+0000 7fca4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.567 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.567+0000 7fca4dffb640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:48:45.567 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.567+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca3c004eb0 con 0x7fca580b4d60 2024-03-25T19:48:45.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.576+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258522180 0 0) 0x7fca58143900 con 0x7fca5814be90 2024-03-25T19:48:45.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.576+0000 7fca4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.576+0000 7fca4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.576+0000 7fca4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.576+0000 7fca4dffb640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:48:45.576 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.576+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca5813b550 con 0x7fca5814be90 2024-03-25T19:48:45.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.577+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3781492589 0 0) 0x7fca3c004eb0 con 0x7fca580b4d60 2024-03-25T19:48:45.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.577+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca58143900 con 0x7fca580b4d60 2024-03-25T19:48:45.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.577+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fca48001a30 con 0x7fca580b4d60 2024-03-25T19:48:45.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.577+0000 7fca4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.577+0000 7fca4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:45.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.577+0000 7fca4dffb640 10 monclient(hunting): dump: 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:45.578 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.577+0000 7fca4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:45.579 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.579+0000 7fca4dffb640 10 client.?.objecter ms_handle_connect 0x7fca580b1730 2024-03-25T19:48:45.579 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.579+0000 7fca4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:45.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.585+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2450988049 0 0) 0x7fca5813b550 con 0x7fca5814be90 2024-03-25T19:48:45.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.585+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca3c004eb0 con 0x7fca5814be90 2024-03-25T19:48:45.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.585+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fca50001bd0 con 0x7fca5814be90 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.585+0000 7fca4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.585+0000 7fca4dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.585+0000 7fca4dffb640 10 monclient(hunting): dump: 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:45.586 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:45.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.588+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188485289 0 0) 0x7fca5813b7a0 con 0x7fca580b1730 2024-03-25T19:48:45.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.588+0000 7fca4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:45.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.588+0000 7fca4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:45.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.588+0000 7fca4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:45.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.588+0000 7fca4dffb640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:48:45.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.588+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca5813b550 con 0x7fca580b1730 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2504759691 0 0) 0x7fca58143900 con 0x7fca580b4d60 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 >> v1:172.21.15.50:6790/0 conn(0x7fca580b1730 legacy=0x7fca58139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 >> v1:172.21.15.50:6789/0 conn(0x7fca5814be90 legacy=0x7fca581d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 1 monclient: found mon.b 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:45.594 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5813bc30 con 0x7fca580b4d60 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 10 monclient: _renew_subs 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca58139140 con 0x7fca580b4d60 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 10 monclient: _renew_subs 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca58130750 con 0x7fca580b4d60 2024-03-25T19:48:45.595 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.594+0000 7fca5f8ba640 1 librados: init done 2024-03-25T19:48:45.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.595+0000 7fca5f8ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:45.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.595+0000 7fca5f8ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:45.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.595+0000 7fca5f8ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:45.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.595+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5813a690 con 0x7fca580b4d60 2024-03-25T19:48:45.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.605+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca48001c70 con 0x7fca580b4d60 2024-03-25T19:48:45.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.605+0000 7fca4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:45.605 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.605+0000 7fca4f7fe640 4 set_mon_vals no callback set 2024-03-25T19:48:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.613+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fca48003a60 con 0x7fca580b4d60 2024-03-25T19:48:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.613+0000 7fca4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.613+0000 7fca4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:45.613 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.613+0000 7fca4dffb640 10 monclient: dump: 2024-03-25T19:48:45.613 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:45.613 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:45.614 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.615+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fca48003c40 con 0x7fca580b4d60 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.615+0000 7fca5d656640 10 monclient: get_auth_request con 0x7fca3c06df00 auth_method 0 2024-03-25T19:48:45.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.616+0000 7fca4dffb640 10 client.4573.objecter ms_handle_connect 0x7fca3c06df00 2024-03-25T19:48:45.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.617+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fca480549a0 con 0x7fca580b4d60 2024-03-25T19:48:45.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.617+0000 7fca4dffb640 10 client.4573.objecter ms_dispatch 0x7fca58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:45.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.617+0000 7fca4dffb640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:45.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.617+0000 7fca4dffb640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:45.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.617+0000 7fca4dffb640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T19:48:45.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.879+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca480502e0 con 0x7fca580b4d60 2024-03-25T19:48:45.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.879+0000 7fca4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:45.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:45.879+0000 7fca4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:46.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.127+0000 7fca5f8ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.127+0000 7fca5f8ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.127+0000 7fca5f8ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:48:46.127 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.127+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca580627b0 con 0x7fca580b4d60 2024-03-25T19:48:46.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.158+0000 7fca4dffb640 1 -- 172.21.15.50:0/3401241294 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fca480523e0 con 0x7fca580b4d60 2024-03-25T19:48:46.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.158+0000 7fca4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:46.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.158+0000 7fca4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:46.158 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:46.158 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:46.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 10 librados: watch_flush enter 2024-03-25T19:48:46.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 10 librados: watch_flush exit 2024-03-25T19:48:46.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T19:48:46.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 10 client.4573.objecter successfully canceled tick 2024-03-25T19:48:46.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 >> v1:172.21.15.76:6800/2572556811 conn(0x7fca3c06df00 legacy=0x7fca3c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:46.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 10 monclient: shutdown 2024-03-25T19:48:46.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:46.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 >> v1:172.21.15.76:6789/0 conn(0x7fca580b4d60 legacy=0x7fca5812fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:46.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 shutdown_connections 2024-03-25T19:48:46.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 >> 172.21.15.50:0/3401241294 conn(0x7fca580617a0 msgr2=0x7fca580b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:46.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 shutdown_connections 2024-03-25T19:48:46.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 1 -- 172.21.15.50:0/3401241294 wait complete. 2024-03-25T19:48:46.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:46.161+0000 7fca5f8ba640 1 librados: shutdown 2024-03-25T19:48:46.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:52.979 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:53.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.086+0000 7f2b5ffa3640 10 monclient: get_monmap_and_config 2024-03-25T19:48:53.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.086+0000 7f2b5ffa3640 10 monclient: build_initial_monmap 2024-03-25T19:48:53.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.086+0000 7f2b5ffa3640 10 monclient: monmap: 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:48:53.086917+0000 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:48:53.086917+0000 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:48:53.086 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:53.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.086+0000 7f2b5ffa3640 1 Processor -- start 2024-03-25T19:48:53.087 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.086+0000 7f2b5ffa3640 1 -- start start 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.086+0000 7f2b5ffa3640 10 monclient: init 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b580b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 10 monclient(hunting): picked mon.noname-a con 0x7f2b580b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 10 monclient(hunting): picked mon.noname-c con 0x7f2b5814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580af940 con 0x7f2b580b1730 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580b0a70 con 0x7f2b5814be90 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b580b0d50 con 0x7f2b580b4d60 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5ffa3640 10 monclient(hunting): authenticate will time out at 5308.351157s 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.087+0000 7f2b5dd3f640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f2b580b4d60 0x7f2b5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:52820/0 (socket says 172.21.15.50:52820) 2024-03-25T19:48:53.088 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.088+0000 7f2b5dd3f640 1 -- 172.21.15.50:0/3550168788 learned_addr learned my addr 172.21.15.50:0/3550168788 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:53.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.124+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478880678 0 0) 0x7f2b580af940 con 0x7f2b580b1730 2024-03-25T19:48:53.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.124+0000 7f2b5cd3d640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.124+0000 7f2b5cd3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.124+0000 7f2b5cd3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.124+0000 7f2b5cd3d640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:48:53.124 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.124+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b40003520 con 0x7f2b580b1730 2024-03-25T19:48:53.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 502603357 0 0) 0x7f2b40003520 con 0x7f2b580b1730 2024-03-25T19:48:53.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 >> v1:172.21.15.50:6790/0 conn(0x7f2b580b4d60 legacy=0x7f2b5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 >> v1:172.21.15.76:6789/0 conn(0x7f2b5814be90 legacy=0x7f2b5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.134 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 10 monclient: _finish_hunting 0 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 1 monclient: found mon.noname-a 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b580b1030 con 0x7f2b580b1730 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 10 monclient: _finish_auth 0 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 10 monclient: _check_auth_tickets 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2b4c001690 con 0x7f2b580b1730 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5cd3d640 10 monclient: dump: 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:53.135 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5ffa3640 5 monclient: authenticate success, global_id 4596 2024-03-25T19:48:53.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.134+0000 7f2b5ffa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:53.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b4c0017f0 con 0x7f2b580b1730 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5cd3d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5cd3d640 1 -- 172.21.15.50:0/3550168788 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2b4c001c00 con 0x7f2b580b1730 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5ffa3640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5ffa3640 4 set_mon_vals no callback set 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5cd3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5cd3d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.157+0000 7f2b5cd3d640 10 monclient: dump: 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:53.157 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:53.158 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:53.158 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:53.159 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.158+0000 7f2b5ffa3640 10 monclient: shutdown 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.158+0000 7f2b5ffa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.158+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3550168788 >> v1:172.21.15.50:6789/0 conn(0x7f2b580b1730 legacy=0x7f2b580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.158+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3550168788 shutdown_connections 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.158+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3550168788 >> 172.21.15.50:0/3550168788 conn(0x7f2b580617a0 msgr2=0x7f2b58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.158+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3550168788 shutdown_connections 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.158+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3550168788 wait complete. 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 10 monclient: build_initial_monmap 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 10 monclient: monmap: 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:53.160 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 1 librados: starting msgr at 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 1 librados: starting objecter 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 1 Processor -- start 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 1 -- start start 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 1 librados: setting wanted keys 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 1 librados: calling monclient init 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 10 monclient: init 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.159+0000 7f2b5ffa3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b580b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 10 monclient(hunting): picked mon.noname-c con 0x7f2b580b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:48:53.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 10 monclient(hunting): picked mon.noname-a con 0x7f2b5814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:48:53.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58143000 con 0x7f2b5814be90 2024-03-25T19:48:53.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58138980 con 0x7f2b580b1730 2024-03-25T19:48:53.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b58138b10 con 0x7f2b580b4d60 2024-03-25T19:48:53.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:53.162 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.160+0000 7f2b5ffa3640 10 monclient(hunting): authenticate will time out at 5308.423780s 2024-03-25T19:48:53.172 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.172+0000 7f2b5dd3f640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f2b580b1730 0x7f2b581db100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:42338/0 (socket says 172.21.15.50:42338) 2024-03-25T19:48:53.172 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.172+0000 7f2b5dd3f640 1 -- 172.21.15.50:0/3895117569 learned_addr learned my addr 172.21.15.50:0/3895117569 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:48:53.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.174+0000 7f2b567fc640 10 client.?.objecter ms_handle_connect 0x7f2b580b1730 2024-03-25T19:48:53.173 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.174+0000 7f2b567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.181 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.182+0000 7f2b567fc640 10 client.?.objecter ms_handle_connect 0x7f2b5814be90 2024-03-25T19:48:53.181 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.182+0000 7f2b567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.197+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961965722 0 0) 0x7f2b58143000 con 0x7f2b5814be90 2024-03-25T19:48:53.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.197+0000 7f2b567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:53.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.197+0000 7f2b567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:53.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.197+0000 7f2b567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:53.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.197+0000 7f2b567fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:48:53.197 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.197+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b44004eb0 con 0x7f2b5814be90 2024-03-25T19:48:53.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.202+0000 7f2b567fc640 10 client.?.objecter ms_handle_connect 0x7f2b580b4d60 2024-03-25T19:48:53.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.202+0000 7f2b567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:53.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.214+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3789884724 0 0) 0x7f2b44004eb0 con 0x7f2b5814be90 2024-03-25T19:48:53.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.215+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b58143000 con 0x7f2b5814be90 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.215+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2b48001b20 con 0x7f2b5814be90 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.215+0000 7f2b567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.215+0000 7f2b567fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.215+0000 7f2b567fc640 10 monclient(hunting): dump: 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:53.215 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.215+0000 7f2b567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3495641866 0 0) 0x7f2b58143000 con 0x7f2b5814be90 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 >> v1:172.21.15.50:6790/0 conn(0x7f2b580b4d60 legacy=0x7f2b581db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 >> v1:172.21.15.76:6789/0 conn(0x7f2b580b1730 legacy=0x7f2b581db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 1 monclient: found mon.a 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:53.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58136160 con 0x7f2b5814be90 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.284+0000 7f2b5ffa3640 5 monclient: authenticate success, global_id 4599 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 10 monclient: _renew_subs 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b581363b0 con 0x7f2b5814be90 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 10 monclient: _renew_subs 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b58142af0 con 0x7f2b5814be90 2024-03-25T19:48:53.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.285+0000 7f2b5ffa3640 1 librados: init done 2024-03-25T19:48:53.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.286+0000 7f2b5ffa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.286+0000 7f2b5ffa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.286+0000 7f2b5ffa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:53.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.286+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b58137340 con 0x7f2b5814be90 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.642+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b48002070 con 0x7f2b5814be90 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.642+0000 7f2b567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.642+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2b48003970 con 0x7f2b5814be90 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.642+0000 7f2b567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.642+0000 7f2b567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.642+0000 7f2b567fc640 10 monclient: dump: 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:48:53.642 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:48:53.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.642+0000 7f2b57fff640 4 set_mon_vals no callback set 2024-03-25T19:48:53.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.643+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f2b48003b90 con 0x7f2b5814be90 2024-03-25T19:48:53.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.644+0000 7f2b5dd3f640 10 monclient: get_auth_request con 0x7f2b4406dd00 auth_method 0 2024-03-25T19:48:53.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.644+0000 7f2b567fc640 10 client.4599.objecter ms_handle_connect 0x7f2b4406dd00 2024-03-25T19:48:53.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.649+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f2b480535f0 con 0x7f2b5814be90 2024-03-25T19:48:53.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.649+0000 7f2b567fc640 10 client.4599.objecter ms_dispatch 0x7f2b58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:53.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.649+0000 7f2b567fc640 3 client.4599.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:53.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.649+0000 7f2b567fc640 3 client.4599.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:53.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.649+0000 7f2b567fc640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-25T19:48:53.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.650+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b4804f100 con 0x7f2b5814be90 2024-03-25T19:48:53.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.650+0000 7f2b567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:53.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.650+0000 7f2b567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:53.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.910+0000 7f2b5ffa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.910+0000 7f2b5ffa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.910+0000 7f2b5ffa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:48:53.910 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.910+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b580627b0 con 0x7f2b5814be90 2024-03-25T19:48:53.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.946+0000 7f2b567fc640 1 -- 172.21.15.50:0/3895117569 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f2b48051030 con 0x7f2b5814be90 2024-03-25T19:48:53.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.946+0000 7f2b567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:53.946 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.946+0000 7f2b567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:53.946 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:48:53.947 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 10 librados: watch_flush enter 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 10 librados: watch_flush exit 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 10 client.4599.objecter successfully canceled tick 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 >> v1:172.21.15.76:6800/2572556811 conn(0x7f2b4406dd00 legacy=0x7f2b440701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 10 monclient: shutdown 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 >> v1:172.21.15.50:6789/0 conn(0x7f2b5814be90 legacy=0x7f2b581e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 shutdown_connections 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.948+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 >> 172.21.15.50:0/3895117569 conn(0x7f2b580617a0 msgr2=0x7f2b580b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.949+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 shutdown_connections 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.949+0000 7f2b5ffa3640 1 -- 172.21.15.50:0/3895117569 wait complete. 2024-03-25T19:48:53.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:48:53.949+0000 7f2b5ffa3640 1 librados: shutdown 2024-03-25T19:48:53.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:00.766 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:00.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.866+0000 7f31bdf20640 10 monclient: get_monmap_and_config 2024-03-25T19:49:00.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.866+0000 7f31bdf20640 10 monclient: build_initial_monmap 2024-03-25T19:49:00.866 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.866+0000 7f31bdf20640 10 monclient: monmap: 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:00.867699+0000 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:00.867699+0000 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:00.867 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:00.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.867+0000 7f31bdf20640 1 Processor -- start 2024-03-25T19:49:00.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.867+0000 7f31bdf20640 1 -- start start 2024-03-25T19:49:00.868 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.867+0000 7f31bdf20640 10 monclient: init 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 10 monclient(hunting): picked mon.noname-b con 0x7f31b80b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 10 monclient(hunting): picked mon.noname-c con 0x7f31b80b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 10 monclient(hunting): picked mon.noname-a con 0x7f31b814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b80af940 con 0x7f31b814be90 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b80b0a70 con 0x7f31b80b1730 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b80b0d50 con 0x7f31b80b4d60 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31bdf20640 10 monclient(hunting): authenticate will time out at 5316.132058s 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31b77fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f31b80b4d60 0x7f31b813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53740/0 (socket says 172.21.15.50:53740) 2024-03-25T19:49:00.869 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.868+0000 7f31b77fe640 1 -- 172.21.15.50:0/2803899785 learned_addr learned my addr 172.21.15.50:0/2803899785 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:00.877 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:21.383 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.886 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:21.392 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.908 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.908+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714249965 0 0) 0x7f31b80af940 con 0x7f31b814be90 2024-03-25T19:49:00.908 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.908+0000 7f31b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:00.908 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.908+0000 7f31b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:00.908 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.908+0000 7f31b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:00.908 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.908+0000 7f31b67fc640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:49:00.908 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.908+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a4003520 con 0x7f31b814be90 2024-03-25T19:49:00.914 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:21.420 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.928 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:21.398 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2852250717 0 0) 0x7f31a4003520 con 0x7f31b814be90 2024-03-25T19:49:00.930 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 >> v1:172.21.15.50:6790/0 conn(0x7f31b80b4d60 legacy=0x7f31b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 >> v1:172.21.15.76:6789/0 conn(0x7f31b80b1730 legacy=0x7f31b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 1 monclient: found mon.noname-a 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b80b1030 con 0x7f31b814be90 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f31b0001a10 con 0x7f31b814be90 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31bdf20640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31bdf20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:00.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31b67fc640 10 monclient: dump: 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:00.932 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.931+0000 7f31bdf20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:00.936 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:21.406 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.957 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:21.462 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b0001b70 con 0x7f31b814be90 2024-03-25T19:49:00.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:00.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31b67fc640 1 -- 172.21.15.50:0/2803899785 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f31b0001f80 con 0x7f31b814be90 2024-03-25T19:49:00.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:00.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 4 set_mon_vals no callback set 2024-03-25T19:49:00.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31b67fc640 10 monclient: dump: 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:00.961 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:00.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 10 monclient: shutdown 2024-03-25T19:49:00.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:00.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 1 -- 172.21.15.50:0/2803899785 >> v1:172.21.15.50:6789/0 conn(0x7f31b814be90 legacy=0x7f31b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 1 -- 172.21.15.50:0/2803899785 shutdown_connections 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 1 -- 172.21.15.50:0/2803899785 >> 172.21.15.50:0/2803899785 conn(0x7f31b80617a0 msgr2=0x7f31b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 1 -- 172.21.15.50:0/2803899785 shutdown_connections 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.961+0000 7f31bdf20640 1 -- 172.21.15.50:0/2803899785 wait complete. 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.962+0000 7f31bdf20640 10 monclient: build_initial_monmap 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.962+0000 7f31bdf20640 10 monclient: monmap: 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:00.963 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:00.964 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:00.964 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:00.964 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:21.467 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.962+0000 7f31bdf20640 1 librados: starting msgr at 2024-03-25T19:49:00.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.962+0000 7f31bdf20640 1 librados: starting objecter 2024-03-25T19:49:00.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.962+0000 7f31bdf20640 1 Processor -- start 2024-03-25T19:49:00.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 1 -- start start 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 1 librados: setting wanted keys 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 1 librados: calling monclient init 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient: init 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient(hunting): picked mon.noname-b con 0x7f31b80b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient(hunting): picked mon.noname-a con 0x7f31b80b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient(hunting): picked mon.noname-c con 0x7f31b814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b813bd10 con 0x7f31b80b4d60 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b81dadb0 con 0x7f31b80b1730 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b8138950 con 0x7f31b814be90 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:00.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.963+0000 7f31bdf20640 10 monclient(hunting): authenticate will time out at 5316.227459s 2024-03-25T19:49:00.967 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:21.434 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:00.982 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.983+0000 7f31b7fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f31b814be90 0x7f31b81db2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53762/0 (socket says 172.21.15.50:53762) 2024-03-25T19:49:00.982 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.983+0000 7f31b7fff640 1 -- 172.21.15.50:0/1962536904 learned_addr learned my addr 172.21.15.50:0/1962536904 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:00.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.991+0000 7f31affff640 10 client.?.objecter ms_handle_connect 0x7f31b814be90 2024-03-25T19:49:00.990 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:00.991+0000 7f31affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:00.998 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:21.504 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.010+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777418071 0 0) 0x7f31b8138950 con 0x7f31b814be90 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.010+0000 7f31affff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.010+0000 7f31affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.010+0000 7f31affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.010+0000 7f31affff640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:49:01.010 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.010+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3198004eb0 con 0x7f31b814be90 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.018+0000 7f31affff640 10 client.?.objecter ms_handle_connect 0x7f31b80b1730 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.018+0000 7f31affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.018+0000 7f31affff640 10 client.?.objecter ms_handle_connect 0x7f31b80b4d60 2024-03-25T19:49:01.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.018+0000 7f31affff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:01.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.025+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1624897768 0 0) 0x7f3198004eb0 con 0x7f31b814be90 2024-03-25T19:49:01.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.025+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31b8138950 con 0x7f31b814be90 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.026+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f31b00020a0 con 0x7f31b814be90 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.026+0000 7f31affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.026+0000 7f31affff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.026+0000 7f31affff640 10 monclient(hunting): dump: 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:01.026 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.027 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:01.027 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:01.027 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:01.027 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:01.027 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:01.027 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.026+0000 7f31affff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:01.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.037+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481331768 0 0) 0x7f31b813bd10 con 0x7f31b80b4d60 2024-03-25T19:49:01.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.037+0000 7f31affff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.037+0000 7f31affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.037+0000 7f31affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.037+0000 7f31affff640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:49:01.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.037+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3198004eb0 con 0x7f31b80b4d60 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.038+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012557258 0 0) 0x7f31b81dadb0 con 0x7f31b80b1730 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.038+0000 7f31affff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.038+0000 7f31affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.038+0000 7f31affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.038+0000 7f31affff640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:49:01.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.038+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b813bd10 con 0x7f31b80b1730 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.055+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4258008832 0 0) 0x7f31b813bd10 con 0x7f31b80b1730 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.055+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31b81dadb0 con 0x7f31b80b1730 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.055+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f31a0001760 con 0x7f31b80b1730 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.055+0000 7f31affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.055+0000 7f31affff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.055+0000 7f31affff640 10 monclient(hunting): dump: 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:01.055 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:01.056 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:01.056 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3452044930 0 0) 0x7f31b8138950 con 0x7f31b814be90 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 >> v1:172.21.15.76:6789/0 conn(0x7f31b80b1730 legacy=0x7f31b8138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 >> v1:172.21.15.50:6789/0 conn(0x7f31b80b4d60 legacy=0x7f31b813a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:01.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 1 monclient: found mon.c 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b813ab60 con 0x7f31b814be90 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 10 monclient: _finish_auth 0 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 5 monclient: authenticate success, global_id 4580 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 10 monclient: _renew_subs 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b81db9e0 con 0x7f31b814be90 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 10 monclient: _renew_subs 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b8139890 con 0x7f31b814be90 2024-03-25T19:49:01.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.070+0000 7f31bdf20640 1 librados: init done 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.071+0000 7f31bdf20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.071+0000 7f31bdf20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.071+0000 7f31bdf20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:01.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.071+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b813acf0 con 0x7f31b814be90 2024-03-25T19:49:01.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31b0003d20 con 0x7f31b814be90 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f31b00040b0 con 0x7f31b814be90 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31affff640 10 monclient: dump: 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:01.396 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:01.397 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:01.397 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:01.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31b5ffb640 4 set_mon_vals no callback set 2024-03-25T19:49:01.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.396+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f31b00042d0 con 0x7f31b814be90 2024-03-25T19:49:01.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.397+0000 7f31b77fe640 10 monclient: get_auth_request con 0x7f319806e020 auth_method 0 2024-03-25T19:49:01.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.397+0000 7f31affff640 10 client.4580.objecter ms_handle_connect 0x7f319806e020 2024-03-25T19:49:01.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.403+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f31b0054c50 con 0x7f31b814be90 2024-03-25T19:49:01.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.403+0000 7f31affff640 10 client.4580.objecter ms_dispatch 0x7f31b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:01.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.403+0000 7f31affff640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:01.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.403+0000 7f31affff640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:01.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.403+0000 7f31affff640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-25T19:49:01.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.404+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31b0050760 con 0x7f31b814be90 2024-03-25T19:49:01.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.404+0000 7f31affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:01.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.404+0000 7f31affff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:01.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.649+0000 7f31bdf20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.649+0000 7f31bdf20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.649+0000 7f31bdf20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:01.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.649+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31b80627b0 con 0x7f31b814be90 2024-03-25T19:49:01.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.680+0000 7f31affff640 1 -- 172.21.15.50:0/1962536904 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f31b0052690 con 0x7f31b814be90 2024-03-25T19:49:01.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.680+0000 7f31affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:01.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.680+0000 7f31affff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:01.680 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:01.680 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:01.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.682+0000 7f31bdf20640 10 librados: watch_flush enter 2024-03-25T19:49:01.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 10 librados: watch_flush exit 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 10 client.4580.objecter successfully canceled tick 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 >> v1:172.21.15.76:6800/2572556811 conn(0x7f319806e020 legacy=0x7f31980704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 10 monclient: shutdown 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 >> v1:172.21.15.50:6790/0 conn(0x7f31b814be90 legacy=0x7f31b81db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 shutdown_connections 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 >> 172.21.15.50:0/1962536904 conn(0x7f31b80617a0 msgr2=0x7f31b813c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:01.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 shutdown_connections 2024-03-25T19:49:01.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 1 -- 172.21.15.50:0/1962536904 wait complete. 2024-03-25T19:49:01.684 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:01.683+0000 7f31bdf20640 1 librados: shutdown 2024-03-25T19:49:01.694 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:08.500 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.617+0000 7f6f015f4640 10 monclient: get_monmap_and_config 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.617+0000 7f6f015f4640 10 monclient: build_initial_monmap 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.617+0000 7f6f015f4640 10 monclient: monmap: 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:08.619217+0000 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:08.619217+0000 2024-03-25T19:49:08.618 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:08.619 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:08.619 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:08.619 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:08.619 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:08.619 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:08.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.618+0000 7f6f015f4640 1 Processor -- start 2024-03-25T19:49:08.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.618+0000 7f6f015f4640 1 -- start start 2024-03-25T19:49:08.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.618+0000 7f6f015f4640 10 monclient: init 2024-03-25T19:49:08.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 10 monclient(hunting): picked mon.noname-a con 0x7f6efc0b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 10 monclient(hunting): picked mon.noname-c con 0x7f6efc0b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 10 monclient(hunting): picked mon.noname-b con 0x7f6efc14be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0af940 con 0x7f6efc0b4d60 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0b0a70 con 0x7f6efc0b1730 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0b0d50 con 0x7f6efc14be90 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.621 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.619+0000 7f6f015f4640 10 monclient(hunting): authenticate will time out at 5323.884166s 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.630+0000 7f6efb7fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f6efc14be90 0x7f6efc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:60854/0 (socket says 172.21.15.50:60854) 2024-03-25T19:49:08.631 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.630+0000 7f6efb7fe640 1 -- 172.21.15.50:0/2490919511 learned_addr learned my addr 172.21.15.50:0/2490919511 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:08.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.636+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807980491 0 0) 0x7f6efc0b0d50 con 0x7f6efc14be90 2024-03-25T19:49:08.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.636+0000 7f6efa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.636+0000 7f6efa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.636+0000 7f6efa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.636+0000 7f6efa7fc640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:49:08.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.636+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee8003520 con 0x7f6efc14be90 2024-03-25T19:49:08.646 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.646+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144074539 0 0) 0x7f6efc0b0a70 con 0x7f6efc0b1730 2024-03-25T19:49:08.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.646+0000 7f6efa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.646+0000 7f6efa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.646+0000 7f6efa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.646+0000 7f6efa7fc640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:49:08.647 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.646+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6efc0b0d50 con 0x7f6efc0b1730 2024-03-25T19:49:08.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 834185355 0 0) 0x7f6ee8003520 con 0x7f6efc14be90 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 >> v1:172.21.15.76:6789/0 conn(0x7f6efc0b1730 legacy=0x7f6efc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 >> v1:172.21.15.50:6789/0 conn(0x7f6efc0b4d60 legacy=0x7f6efc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 1 monclient: found mon.noname-b 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc0b1030 con 0x7f6efc14be90 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6ef4001a10 con 0x7f6efc14be90 2024-03-25T19:49:08.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6efa7fc640 10 monclient: dump: 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:08.650 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.647+0000 7f6f015f4640 5 monclient: authenticate success, global_id 4583 2024-03-25T19:49:08.651 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.648+0000 7f6f015f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:08.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ef4001b70 con 0x7f6efc14be90 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6efa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6efa7fc640 1 -- 172.21.15.50:0/2490919511 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6ef4001f80 con 0x7f6efc14be90 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6efa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6efa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6efa7fc640 10 monclient: dump: 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:08.664 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:08.665 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:08.665 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:08.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6f015f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:08.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.663+0000 7f6f015f4640 4 set_mon_vals no callback set 2024-03-25T19:49:08.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.664+0000 7f6f015f4640 10 monclient: shutdown 2024-03-25T19:49:08.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.664+0000 7f6f015f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:08.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.664+0000 7f6f015f4640 1 -- 172.21.15.50:0/2490919511 >> v1:172.21.15.50:6790/0 conn(0x7f6efc14be90 legacy=0x7f6efc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.664+0000 7f6f015f4640 1 -- 172.21.15.50:0/2490919511 shutdown_connections 2024-03-25T19:49:08.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.664+0000 7f6f015f4640 1 -- 172.21.15.50:0/2490919511 >> 172.21.15.50:0/2490919511 conn(0x7f6efc0617a0 msgr2=0x7f6efc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:08.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.664+0000 7f6f015f4640 1 -- 172.21.15.50:0/2490919511 shutdown_connections 2024-03-25T19:49:08.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.664+0000 7f6f015f4640 1 -- 172.21.15.50:0/2490919511 wait complete. 2024-03-25T19:49:08.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.665+0000 7f6f015f4640 10 monclient: build_initial_monmap 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.665+0000 7f6f015f4640 10 monclient: monmap: 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 librados: starting msgr at 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 librados: starting objecter 2024-03-25T19:49:08.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 Processor -- start 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 -- start start 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 librados: setting wanted keys 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 librados: calling monclient init 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient: init 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient(hunting): picked mon.noname-b con 0x7f6efc0b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient(hunting): picked mon.noname-c con 0x7f6efc0b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient(hunting): picked mon.noname-a con 0x7f6efc14be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc137220 con 0x7f6efc14be90 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc145330 con 0x7f6efc0b1730 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc13e750 con 0x7f6efc0b4d60 2024-03-25T19:49:08.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.666+0000 7f6f015f4640 10 monclient(hunting): authenticate will time out at 5323.931655s 2024-03-25T19:49:08.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.670+0000 7f6efb7fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f6efc14be90 0x7f6efc144c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:41830/0 (socket says 172.21.15.50:41830) 2024-03-25T19:49:08.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.670+0000 7f6efb7fe640 1 -- 172.21.15.50:0/186526445 learned_addr learned my addr 172.21.15.50:0/186526445 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:08.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.672+0000 7f6ef37fe640 10 client.?.objecter ms_handle_connect 0x7f6efc14be90 2024-03-25T19:49:08.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.672+0000 7f6ef37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.687+0000 7f6ef37fe640 10 client.?.objecter ms_handle_connect 0x7f6efc0b4d60 2024-03-25T19:49:08.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.687+0000 7f6ef37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.690+0000 7f6ef37fe640 10 client.?.objecter ms_handle_connect 0x7f6efc0b1730 2024-03-25T19:49:08.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.690+0000 7f6ef37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.693+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161317398 0 0) 0x7f6efc137220 con 0x7f6efc14be90 2024-03-25T19:49:08.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.693+0000 7f6ef37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.693+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.693+0000 7f6ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.693+0000 7f6ef37fe640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:49:08.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.693+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6edc004eb0 con 0x7f6efc14be90 2024-03-25T19:49:08.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.702+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330900344 0 0) 0x7f6efc145330 con 0x7f6efc0b1730 2024-03-25T19:49:08.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.702+0000 7f6ef37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.702+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.702+0000 7f6ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.702+0000 7f6ef37fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.702+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6efc137220 con 0x7f6efc0b1730 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.703+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672993779 0 0) 0x7f6efc13e750 con 0x7f6efc0b4d60 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.703+0000 7f6ef37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.703+0000 7f6ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.703+0000 7f6ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.703+0000 7f6ef37fe640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:49:08.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.703+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6efc145330 con 0x7f6efc0b4d60 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.713+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550953961 0 0) 0x7f6edc004eb0 con 0x7f6efc14be90 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.713+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6efc13e750 con 0x7f6efc14be90 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.713+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6ef4001ec0 con 0x7f6efc14be90 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.713+0000 7f6ef37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.713+0000 7f6ef37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.713+0000 7f6ef37fe640 10 monclient(hunting): dump: 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:08.714 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:08.715 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.713+0000 7f6ef37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.717+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2723554492 0 0) 0x7f6efc145330 con 0x7f6efc0b4d60 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.717+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6edc004eb0 con 0x7f6efc0b4d60 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.717+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6ee4001930 con 0x7f6efc0b4d60 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.717+0000 7f6ef37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.717+0000 7f6ef37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.717+0000 7f6ef37fe640 10 monclient(hunting): dump: 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.718 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:08.719 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:08.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1388197890 0 0) 0x7f6edc004eb0 con 0x7f6efc0b4d60 2024-03-25T19:49:08.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 >> v1:172.21.15.76:6789/0 conn(0x7f6efc0b1730 legacy=0x7f6efc1dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 >> v1:172.21.15.50:6789/0 conn(0x7f6efc14be90 legacy=0x7f6efc144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 1 monclient: found mon.c 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc1dba10 con 0x7f6efc0b4d60 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6ef37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 10 monclient: _renew_subs 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc13dba0 con 0x7f6efc0b4d60 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:08.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 10 monclient: _renew_subs 2024-03-25T19:49:08.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:08.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc13ee50 con 0x7f6efc0b4d60 2024-03-25T19:49:08.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.733+0000 7f6f015f4640 1 librados: init done 2024-03-25T19:49:08.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.734+0000 7f6f015f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.734+0000 7f6f015f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.734+0000 7f6f015f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:08.736 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.734+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6efc13f100 con 0x7f6efc0b4d60 2024-03-25T19:49:08.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.741+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ee40035f0 con 0x7f6efc0b4d60 2024-03-25T19:49:08.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.741+0000 7f6ef37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:08.742 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.741+0000 7f6ef97fa640 4 set_mon_vals no callback set 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.745+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6ee4003900 con 0x7f6efc0b4d60 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.745+0000 7f6ef37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.745+0000 7f6ef37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:08.745+0000 7f6ef37fe640 10 monclient: dump: 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:08.746 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:08.747 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:08.747 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:08.747 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:09.058 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.058+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f6ee40198d0 con 0x7f6efc0b4d60 2024-03-25T19:49:09.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.058+0000 7f6efaffd640 10 monclient: get_auth_request con 0x7f6edc06dfb0 auth_method 0 2024-03-25T19:49:09.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.059+0000 7f6ef37fe640 10 client.4586.objecter ms_handle_connect 0x7f6edc06dfb0 2024-03-25T19:49:09.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f6ee40535f0 con 0x7f6efc0b4d60 2024-03-25T19:49:09.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 10 client.4586.objecter ms_dispatch 0x7f6efc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:09.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:09.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:09.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T19:49:09.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ee404ef30 con 0x7f6efc0b4d60 2024-03-25T19:49:09.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:09.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.069+0000 7f6ef37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:09.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.289+0000 7f6f015f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.289+0000 7f6f015f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.289+0000 7f6f015f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:09.290 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.289+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6efc0627b0 con 0x7f6efc0b4d60 2024-03-25T19:49:09.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.355+0000 7f6ef37fe640 1 -- 172.21.15.50:0/186526445 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f6ee4051030 con 0x7f6efc0b4d60 2024-03-25T19:49:09.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.355+0000 7f6ef37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.355+0000 7f6ef37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:09.356 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:09.356 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:09.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 10 librados: watch_flush enter 2024-03-25T19:49:09.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 10 librados: watch_flush exit 2024-03-25T19:49:09.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T19:49:09.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 10 client.4586.objecter successfully canceled tick 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 >> v1:172.21.15.76:6800/2572556811 conn(0x7f6edc06dfb0 legacy=0x7f6edc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 10 monclient: shutdown 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 >> v1:172.21.15.50:6790/0 conn(0x7f6efc0b4d60 legacy=0x7f6efc136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 shutdown_connections 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 >> 172.21.15.50:0/186526445 conn(0x7f6efc0617a0 msgr2=0x7f6efc0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 shutdown_connections 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 1 -- 172.21.15.50:0/186526445 wait complete. 2024-03-25T19:49:09.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:09.357+0000 7f6f015f4640 1 librados: shutdown 2024-03-25T19:49:09.368 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:16.174 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.292+0000 7fee23e4e640 10 monclient: get_monmap_and_config 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.292+0000 7fee23e4e640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.292+0000 7fee23e4e640 10 monclient: monmap: 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:16.293170+0000 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:16.293170+0000 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.292 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:16.293 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:16.293 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:16.293 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:16.293 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:16.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.293+0000 7fee23e4e640 1 Processor -- start 2024-03-25T19:49:16.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.293+0000 7fee23e4e640 1 -- start start 2024-03-25T19:49:16.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.293+0000 7fee23e4e640 10 monclient: init 2024-03-25T19:49:16.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.293+0000 7fee23e4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:16.294 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 10 monclient(hunting): picked mon.noname-b con 0x7fee1c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:16.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 10 monclient(hunting): picked mon.noname-c con 0x7fee1c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:16.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 10 monclient(hunting): picked mon.noname-a con 0x7fee1c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:16.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c142520 con 0x7fee1c1430c0 2024-03-25T19:49:16.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c0b28a0 con 0x7fee1c13f710 2024-03-25T19:49:16.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c0b2a30 con 0x7fee1c0a98f0 2024-03-25T19:49:16.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.294+0000 7fee23e4e640 10 monclient(hunting): authenticate will time out at 5331.557958s 2024-03-25T19:49:16.308 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.308+0000 7fee213e9640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fee1c13f710 0x7fee1c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:43754/0 (socket says 172.21.15.50:43754) 2024-03-25T19:49:16.308 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.308+0000 7fee213e9640 1 -- 172.21.15.50:0/2952557463 learned_addr learned my addr 172.21.15.50:0/2952557463 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:16.326 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.327+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501367366 0 0) 0x7fee1c0b28a0 con 0x7fee1c13f710 2024-03-25T19:49:16.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.327+0000 7fee20be8640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.327+0000 7fee20be8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.327+0000 7fee20be8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.327+0000 7fee20be8640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:49:16.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.327+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee08003520 con 0x7fee1c13f710 2024-03-25T19:49:16.331 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.332+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403047252 0 0) 0x7fee1c142520 con 0x7fee1c1430c0 2024-03-25T19:49:16.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.332+0000 7fee20be8640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.332+0000 7fee20be8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.332+0000 7fee20be8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.332+0000 7fee20be8640 10 monclient(hunting): my global_id is 4614 2024-03-25T19:49:16.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.332+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c0b28a0 con 0x7fee1c1430c0 2024-03-25T19:49:16.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.342+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967965017 0 0) 0x7fee1c0b2a30 con 0x7fee1c0a98f0 2024-03-25T19:49:16.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.342+0000 7fee20be8640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.341 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.342+0000 7fee20be8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.342+0000 7fee20be8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.342+0000 7fee20be8640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:49:16.342 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.342+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c142520 con 0x7fee1c0a98f0 2024-03-25T19:49:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3596028857 0 0) 0x7fee1c0b28a0 con 0x7fee1c1430c0 2024-03-25T19:49:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 >> v1:172.21.15.50:6790/0 conn(0x7fee1c0a98f0 legacy=0x7fee1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 >> v1:172.21.15.76:6789/0 conn(0x7fee1c13f710 legacy=0x7fee1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 1 monclient: found mon.noname-a 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c0b44c0 con 0x7fee1c1430c0 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fee140019f0 con 0x7fee1c1430c0 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee20be8640 10 monclient: dump: 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.352 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee23e4e640 5 monclient: authenticate success, global_id 4614 2024-03-25T19:49:16.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.351+0000 7fee23e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:16.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee14001b50 con 0x7fee1c1430c0 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee20be8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee20be8640 1 -- 172.21.15.50:0/2952557463 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fee14001f60 con 0x7fee1c1430c0 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee20be8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee20be8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee20be8640 10 monclient: dump: 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:16.397 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:16.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee23e4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:16.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee23e4e640 4 set_mon_vals no callback set 2024-03-25T19:49:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee23e4e640 10 monclient: shutdown 2024-03-25T19:49:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee23e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee23e4e640 1 -- 172.21.15.50:0/2952557463 >> v1:172.21.15.50:6789/0 conn(0x7fee1c1430c0 legacy=0x7fee1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee23e4e640 1 -- 172.21.15.50:0/2952557463 shutdown_connections 2024-03-25T19:49:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.397+0000 7fee23e4e640 1 -- 172.21.15.50:0/2952557463 >> 172.21.15.50:0/2952557463 conn(0x7fee1c0b7ea0 msgr2=0x7fee1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:16.401 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.398+0000 7fee23e4e640 1 -- 172.21.15.50:0/2952557463 shutdown_connections 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.398+0000 7fee23e4e640 1 -- 172.21.15.50:0/2952557463 wait complete. 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.398+0000 7fee23e4e640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.398+0000 7fee23e4e640 10 monclient: monmap: 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.402 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:16.403 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:16.403 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:16.403 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:16.403 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:16.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 librados: starting msgr at 2024-03-25T19:49:16.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 librados: starting objecter 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 Processor -- start 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 -- start start 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 librados: setting wanted keys 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 librados: calling monclient init 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient: init 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient(hunting): picked mon.noname-a con 0x7fee1c0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient(hunting): picked mon.noname-c con 0x7fee1c13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient(hunting): picked mon.noname-b con 0x7fee1c1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:16.404 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c1d48f0 con 0x7fee1c0a98f0 2024-03-25T19:49:16.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c1397d0 con 0x7fee1c1430c0 2024-03-25T19:49:16.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c1d4b50 con 0x7fee1c13f710 2024-03-25T19:49:16.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.399+0000 7fee23e4e640 10 monclient(hunting): authenticate will time out at 5331.663647s 2024-03-25T19:49:16.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.404+0000 7fee213e9640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fee1c13f710 0x7fee1c0aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:37226/0 (socket says 172.21.15.50:37226) 2024-03-25T19:49:16.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.404+0000 7fee213e9640 1 -- 172.21.15.50:0/1327690910 learned_addr learned my addr 172.21.15.50:0/1327690910 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:16.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.405+0000 7fee127fc640 10 client.?.objecter ms_handle_connect 0x7fee1c13f710 2024-03-25T19:49:16.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.405+0000 7fee127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.413+0000 7fee127fc640 10 client.?.objecter ms_handle_connect 0x7fee1c0a98f0 2024-03-25T19:49:16.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.413+0000 7fee127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.421+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849590775 0 0) 0x7fee1c1d4b50 con 0x7fee1c13f710 2024-03-25T19:49:16.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.421+0000 7fee127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.421+0000 7fee127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.421+0000 7fee127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.421+0000 7fee127fc640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:49:16.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.421+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee00004eb0 con 0x7fee1c13f710 2024-03-25T19:49:16.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.431+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406706614 0 0) 0x7fee00004eb0 con 0x7fee1c13f710 2024-03-25T19:49:16.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.431+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee1c1d4b50 con 0x7fee1c13f710 2024-03-25T19:49:16.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.431+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fee0c001950 con 0x7fee1c13f710 2024-03-25T19:49:16.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.431+0000 7fee127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.431+0000 7fee127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:16.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.431+0000 7fee127fc640 10 monclient(hunting): dump: 2024-03-25T19:49:16.431 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:16.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.431+0000 7fee127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:16.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.432+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039692722 0 0) 0x7fee1c1d48f0 con 0x7fee1c0a98f0 2024-03-25T19:49:16.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.432+0000 7fee127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.432+0000 7fee127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.432+0000 7fee127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.432+0000 7fee127fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:49:16.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.432+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee00004eb0 con 0x7fee1c0a98f0 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.435+0000 7fee127fc640 10 client.?.objecter ms_handle_connect 0x7fee1c1430c0 2024-03-25T19:49:16.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.435+0000 7fee127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.440+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380226878 0 0) 0x7fee1c1397d0 con 0x7fee1c1430c0 2024-03-25T19:49:16.439 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.440+0000 7fee127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.440+0000 7fee127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.440+0000 7fee127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.440+0000 7fee127fc640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:49:16.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.440+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c1d48f0 con 0x7fee1c1430c0 2024-03-25T19:49:16.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3918611722 0 0) 0x7fee1c1d4b50 con 0x7fee1c13f710 2024-03-25T19:49:16.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 >> v1:172.21.15.76:6789/0 conn(0x7fee1c1430c0 legacy=0x7fee1c1d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 >> v1:172.21.15.50:6789/0 conn(0x7fee1c0a98f0 legacy=0x7fee1c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.445 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 1 monclient: found mon.c 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c0b16b0 con 0x7fee1c13f710 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 5 monclient: authenticate success, global_id 4592 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 10 monclient: _renew_subs 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c0b0c10 con 0x7fee1c13f710 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:16.446 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 10 monclient: _renew_subs 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c0b0610 con 0x7fee1c13f710 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.445+0000 7fee23e4e640 1 librados: init done 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.447+0000 7fee23e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.447+0000 7fee23e4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.447+0000 7fee23e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:16.447 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.447+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c1d4ce0 con 0x7fee1c13f710 2024-03-25T19:49:16.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.760+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee0c0047e0 con 0x7fee1c13f710 2024-03-25T19:49:16.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.760+0000 7fee127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.760+0000 7fee13fff640 4 set_mon_vals no callback set 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.760+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fee0c004b70 con 0x7fee1c13f710 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.760+0000 7fee127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.760+0000 7fee127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.760+0000 7fee127fc640 10 monclient: dump: 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:16.761 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:16.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.761+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fee0c004d90 con 0x7fee1c13f710 2024-03-25T19:49:16.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.762+0000 7fee21bea640 10 monclient: get_auth_request con 0x7fee0006de10 auth_method 0 2024-03-25T19:49:16.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.762+0000 7fee127fc640 10 client.4592.objecter ms_handle_connect 0x7fee0006de10 2024-03-25T19:49:16.767 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.767+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fee0c054890 con 0x7fee1c13f710 2024-03-25T19:49:16.767 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.767+0000 7fee127fc640 10 client.4592.objecter ms_dispatch 0x7fee1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:16.767 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.767+0000 7fee127fc640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:16.767 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.767+0000 7fee127fc640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:16.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.768+0000 7fee127fc640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T19:49:16.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.768+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee0c0503a0 con 0x7fee1c13f710 2024-03-25T19:49:16.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.768+0000 7fee127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.768 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:16.768+0000 7fee127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:17.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.025+0000 7fee23e4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.025+0000 7fee23e4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.025+0000 7fee23e4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:17.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.025+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee1c0627b0 con 0x7fee1c13f710 2024-03-25T19:49:17.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.067+0000 7fee127fc640 1 -- 172.21.15.50:0/1327690910 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fee0c0522d0 con 0x7fee1c13f710 2024-03-25T19:49:17.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.067+0000 7fee127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.067+0000 7fee127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:17.067 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:17.067 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:17.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 10 librados: watch_flush enter 2024-03-25T19:49:17.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 10 librados: watch_flush exit 2024-03-25T19:49:17.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T19:49:17.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 10 client.4592.objecter successfully canceled tick 2024-03-25T19:49:17.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 >> v1:172.21.15.76:6800/2572556811 conn(0x7fee0006de10 legacy=0x7fee000702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:17.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 10 monclient: shutdown 2024-03-25T19:49:17.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:17.070 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 >> v1:172.21.15.50:6790/0 conn(0x7fee1c13f710 legacy=0x7fee1c0aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:17.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 shutdown_connections 2024-03-25T19:49:17.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 >> 172.21.15.50:0/1327690910 conn(0x7fee1c0b7ea0 msgr2=0x7fee1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:17.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 shutdown_connections 2024-03-25T19:49:17.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 1 -- 172.21.15.50:0/1327690910 wait complete. 2024-03-25T19:49:17.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:17.069+0000 7fee23e4e640 1 librados: shutdown 2024-03-25T19:49:17.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:20.777 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:41.283 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.795 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:41.301 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.830 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:08:41.336 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.854 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:41.324 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.857 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:41.362 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.858 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:41.329 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.863 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:41.369 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.887 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:08:41.357 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:20.895 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:08:41.400 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.888 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:24.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.005+0000 7efdde7a3640 10 monclient: get_monmap_and_config 2024-03-25T19:49:24.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.005+0000 7efdde7a3640 10 monclient: build_initial_monmap 2024-03-25T19:49:24.004 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.005+0000 7efdde7a3640 10 monclient: monmap: 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:24.005958+0000 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:24.005958+0000 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:24.005 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:24.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.005+0000 7efdde7a3640 1 Processor -- start 2024-03-25T19:49:24.006 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 1 -- start start 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient: init 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient(hunting): picked mon.noname-c con 0x7efdd80a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient(hunting): picked mon.noname-a con 0x7efdd813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient(hunting): picked mon.noname-b con 0x7efdd81430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd8142520 con 0x7efdd813f710 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd80b28a0 con 0x7efdd80a98f0 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd80b2a30 con 0x7efdd81430c0 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:24.007 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.006+0000 7efdde7a3640 10 monclient(hunting): authenticate will time out at 5339.270480s 2024-03-25T19:49:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.011+0000 7efdd7fff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efdd80a98f0 0x7efdd80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:43350/0 (socket says 172.21.15.50:43350) 2024-03-25T19:49:24.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.011+0000 7efdd7fff640 1 -- 172.21.15.50:0/4126723255 learned_addr learned my addr 172.21.15.50:0/4126723255 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:24.029 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.030+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969763025 0 0) 0x7efdd80b28a0 con 0x7efdd80a98f0 2024-03-25T19:49:24.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.030+0000 7efdd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.030+0000 7efdd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.030+0000 7efdd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.030+0000 7efdd6ffd640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:49:24.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.030+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc0003520 con 0x7efdd80a98f0 2024-03-25T19:49:24.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.034+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537920731 0 0) 0x7efdd8142520 con 0x7efdd813f710 2024-03-25T19:49:24.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.034+0000 7efdd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.034+0000 7efdd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.034+0000 7efdd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.034+0000 7efdd6ffd640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:49:24.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.034+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd80b28a0 con 0x7efdd813f710 2024-03-25T19:49:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.035+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486574579 0 0) 0x7efdd80b2a30 con 0x7efdd81430c0 2024-03-25T19:49:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.035+0000 7efdd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.035+0000 7efdd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.035+0000 7efdd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.035+0000 7efdd6ffd640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:49:24.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.035+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd8142520 con 0x7efdd81430c0 2024-03-25T19:49:24.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708841991 0 0) 0x7efdc0003520 con 0x7efdd80a98f0 2024-03-25T19:49:24.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 >> v1:172.21.15.50:6790/0 conn(0x7efdd81430c0 legacy=0x7efdd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 >> v1:172.21.15.50:6789/0 conn(0x7efdd813f710 legacy=0x7efdd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd80b44c0 con 0x7efdd80a98f0 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efdc8001690 con 0x7efdd80a98f0 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.036+0000 7efdd6ffd640 10 monclient: dump: 2024-03-25T19:49:24.037 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.037+0000 7efdde7a3640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:49:24.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.037+0000 7efdde7a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc80017f0 con 0x7efdd80a98f0 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdd6ffd640 1 -- 172.21.15.50:0/4126723255 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efdc8001c00 con 0x7efdd80a98f0 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdde7a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdde7a3640 4 set_mon_vals no callback set 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.050+0000 7efdd6ffd640 10 monclient: dump: 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.050 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.051 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:24.051 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:24.051 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:24.051 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:24.051 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.051+0000 7efdde7a3640 10 monclient: shutdown 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.051+0000 7efdde7a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.051+0000 7efdde7a3640 1 -- 172.21.15.50:0/4126723255 >> v1:172.21.15.76:6789/0 conn(0x7efdd80a98f0 legacy=0x7efdd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.051+0000 7efdde7a3640 1 -- 172.21.15.50:0/4126723255 shutdown_connections 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.051+0000 7efdde7a3640 1 -- 172.21.15.50:0/4126723255 >> 172.21.15.50:0/4126723255 conn(0x7efdd80b7ea0 msgr2=0x7efdd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.051+0000 7efdde7a3640 1 -- 172.21.15.50:0/4126723255 shutdown_connections 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.051+0000 7efdde7a3640 1 -- 172.21.15.50:0/4126723255 wait complete. 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.052+0000 7efdde7a3640 10 monclient: build_initial_monmap 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.052+0000 7efdde7a3640 10 monclient: monmap: 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:24.052 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.052+0000 7efdde7a3640 1 librados: starting msgr at 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.052+0000 7efdde7a3640 1 librados: starting objecter 2024-03-25T19:49:24.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 1 Processor -- start 2024-03-25T19:49:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 1 -- start start 2024-03-25T19:49:24.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 1 librados: setting wanted keys 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 1 librados: calling monclient init 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 10 monclient: init 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 10 monclient(hunting): picked mon.noname-b con 0x7efdd80a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.053+0000 7efdde7a3640 10 monclient(hunting): picked mon.noname-a con 0x7efdd813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.054+0000 7efdde7a3640 10 monclient(hunting): picked mon.noname-c con 0x7efdd81430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.054+0000 7efdde7a3640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd8132ba0 con 0x7efdd813f710 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.054+0000 7efdde7a3640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd8138150 con 0x7efdd80a98f0 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.054+0000 7efdde7a3640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd8139510 con 0x7efdd81430c0 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.054+0000 7efdde7a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:24.055 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.054+0000 7efdde7a3640 10 monclient(hunting): authenticate will time out at 5339.317929s 2024-03-25T19:49:24.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.056+0000 7efdd7fff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efdd80a98f0 0x7efdd8137190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:43390/0 (socket says 172.21.15.50:43390) 2024-03-25T19:49:24.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.056+0000 7efdd7fff640 1 -- 172.21.15.50:0/938869766 learned_addr learned my addr 172.21.15.50:0/938869766 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:24.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.056+0000 7efdd4ff9640 10 client.?.objecter ms_handle_connect 0x7efdd80a98f0 2024-03-25T19:49:24.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.056+0000 7efdd4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.065+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382592736 0 0) 0x7efdd8138150 con 0x7efdd80a98f0 2024-03-25T19:49:24.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.065+0000 7efdd4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.065+0000 7efdd4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.065+0000 7efdd4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.065+0000 7efdd4ff9640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:49:24.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.065+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc4004eb0 con 0x7efdd80a98f0 2024-03-25T19:49:24.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.069+0000 7efdd4ff9640 10 client.?.objecter ms_handle_connect 0x7efdd813f710 2024-03-25T19:49:24.069 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.069+0000 7efdd4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.072+0000 7efdd4ff9640 10 client.?.objecter ms_handle_connect 0x7efdd81430c0 2024-03-25T19:49:24.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.072+0000 7efdd4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.075+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688360830 0 0) 0x7efdc4004eb0 con 0x7efdd80a98f0 2024-03-25T19:49:24.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.075+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdd8138150 con 0x7efdd80a98f0 2024-03-25T19:49:24.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.075+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efdc8001bc0 con 0x7efdd80a98f0 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.075+0000 7efdd4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.075+0000 7efdd4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.075+0000 7efdd4ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:24.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.075+0000 7efdd4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:24.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3236525653 0 0) 0x7efdd8138150 con 0x7efdd80a98f0 2024-03-25T19:49:24.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 >> v1:172.21.15.50:6790/0 conn(0x7efdd81430c0 legacy=0x7efdd81e27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.082 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 >> v1:172.21.15.50:6789/0 conn(0x7efdd813f710 legacy=0x7efdd8135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 1 monclient: found mon.b 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd81397f0 con 0x7efdd80a98f0 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdd4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 5 monclient: authenticate success, global_id 4600 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 10 monclient: _renew_subs 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd8138600 con 0x7efdd80a98f0 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 10 monclient: _renew_subs 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:24.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd8135950 con 0x7efdd80a98f0 2024-03-25T19:49:24.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.082+0000 7efdde7a3640 1 librados: init done 2024-03-25T19:49:24.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.083+0000 7efdde7a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.083+0000 7efdde7a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.083+0000 7efdde7a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:24.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.083+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd80627b0 con 0x7efdd80a98f0 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.350+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc80011f0 con 0x7efdd80a98f0 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.350+0000 7efdd4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.350+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7efdc8004c70 con 0x7efdd80a98f0 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.350+0000 7efdd4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.350+0000 7efdd4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.350+0000 7efdd4ff9640 10 monclient: dump: 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:24.350 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.351 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:24.351 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:24.351 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:24.351 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:24.351 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:24.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.350+0000 7efdd67fc640 4 set_mon_vals no callback set 2024-03-25T19:49:24.351 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.351+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7efdc8004e90 con 0x7efdd80a98f0 2024-03-25T19:49:24.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.351+0000 7efdd77fe640 10 monclient: get_auth_request con 0x7efdc406dc00 auth_method 0 2024-03-25T19:49:24.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.351+0000 7efdd4ff9640 10 client.4600.objecter ms_handle_connect 0x7efdc406dc00 2024-03-25T19:49:24.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.357+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7efdc80549c0 con 0x7efdd80a98f0 2024-03-25T19:49:24.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.357+0000 7efdd4ff9640 10 client.4600.objecter ms_dispatch 0x7efdd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:24.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.357+0000 7efdd4ff9640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:24.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.357+0000 7efdd4ff9640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:24.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.357+0000 7efdd4ff9640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T19:49:24.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.358+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdc8050300 con 0x7efdd80a98f0 2024-03-25T19:49:24.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.358+0000 7efdd4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.358+0000 7efdd4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.623+0000 7efdde7a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.623+0000 7efdde7a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.623+0000 7efdde7a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:24.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.623+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdd813dcf0 con 0x7efdd80a98f0 2024-03-25T19:49:24.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.649+0000 7efdd4ff9640 1 -- 172.21.15.50:0/938869766 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7efdc8052400 con 0x7efdd80a98f0 2024-03-25T19:49:24.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.649+0000 7efdd4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.649+0000 7efdd4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:24.649 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:24.649 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.651+0000 7efdde7a3640 10 librados: watch_flush enter 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.651+0000 7efdde7a3640 10 librados: watch_flush exit 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.651+0000 7efdde7a3640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.651+0000 7efdde7a3640 10 client.4600.objecter successfully canceled tick 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 >> v1:172.21.15.76:6800/2572556811 conn(0x7efdc406dc00 legacy=0x7efdc40700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 10 monclient: shutdown 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 >> v1:172.21.15.76:6789/0 conn(0x7efdd80a98f0 legacy=0x7efdd8137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 shutdown_connections 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 >> 172.21.15.50:0/938869766 conn(0x7efdd80b7ea0 msgr2=0x7efdd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 shutdown_connections 2024-03-25T19:49:24.652 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 1 -- 172.21.15.50:0/938869766 wait complete. 2024-03-25T19:49:24.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:24.652+0000 7efdde7a3640 1 librados: shutdown 2024-03-25T19:49:24.664 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:31.470 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:31.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.587+0000 7f0912f59640 10 monclient: get_monmap_and_config 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.587+0000 7f0912f59640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.587+0000 7f0912f59640 10 monclient: monmap: 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:31.588153+0000 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:31.588153+0000 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:31.587 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 1 Processor -- start 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 1 -- start start 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 10 monclient: init 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 10 monclient(hunting): picked mon.noname-c con 0x7f090c0b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 10 monclient(hunting): picked mon.noname-b con 0x7f090c0b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 10 monclient(hunting): picked mon.noname-a con 0x7f090c14be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.588+0000 7f0912f59640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c0af940 con 0x7f090c14be90 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.589+0000 7f0912f59640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c0b0a70 con 0x7f090c0b4d60 2024-03-25T19:49:31.589 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.589+0000 7f0912f59640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c0b0d50 con 0x7f090c0b1730 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.589+0000 7f0912f59640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.590 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.589+0000 7f0912f59640 10 monclient(hunting): authenticate will time out at 5346.852724s 2024-03-25T19:49:31.596 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.597+0000 7f090bfff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f090c0b1730 0x7f090c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:51324/0 (socket says 172.21.15.50:51324) 2024-03-25T19:49:31.597 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.597+0000 7f090bfff640 1 -- 172.21.15.50:0/748501513 learned_addr learned my addr 172.21.15.50:0/748501513 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:31.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.610+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118985401 0 0) 0x7f090c0b0d50 con 0x7f090c0b1730 2024-03-25T19:49:31.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.610+0000 7f090b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.610+0000 7f090b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.610+0000 7f090b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.610+0000 7f090b7fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:49:31.610 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.610+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08fc003520 con 0x7f090c0b1730 2024-03-25T19:49:31.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 495689381 0 0) 0x7f08fc003520 con 0x7f090c0b1730 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 >> v1:172.21.15.76:6789/0 conn(0x7f090c0b4d60 legacy=0x7f090c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 >> v1:172.21.15.50:6789/0 conn(0x7f090c14be90 legacy=0x7f090c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c0b1030 con 0x7f090c0b1730 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f08f8001690 con 0x7f090c0b1730 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f090b7fe640 10 monclient: dump: 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f0912f59640 5 monclient: authenticate success, global_id 4601 2024-03-25T19:49:31.618 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.617+0000 7f0912f59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08f80017f0 con 0x7f090c0b1730 2024-03-25T19:49:31.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f090b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f090b7fe640 1 -- 172.21.15.50:0/748501513 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f08f8001c40 con 0x7f090c0b1730 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f090b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f090b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f090b7fe640 10 monclient: dump: 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:31.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f0912f59640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:31.638 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f0912f59640 4 set_mon_vals no callback set 2024-03-25T19:49:31.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f0912f59640 10 monclient: shutdown 2024-03-25T19:49:31.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f0912f59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:31.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.637+0000 7f0912f59640 1 -- 172.21.15.50:0/748501513 >> v1:172.21.15.50:6790/0 conn(0x7f090c0b1730 legacy=0x7f090c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.638+0000 7f0912f59640 1 -- 172.21.15.50:0/748501513 shutdown_connections 2024-03-25T19:49:31.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.638+0000 7f0912f59640 1 -- 172.21.15.50:0/748501513 >> 172.21.15.50:0/748501513 conn(0x7f090c0617a0 msgr2=0x7f090c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.638+0000 7f0912f59640 1 -- 172.21.15.50:0/748501513 shutdown_connections 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.638+0000 7f0912f59640 1 -- 172.21.15.50:0/748501513 wait complete. 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.638+0000 7f0912f59640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.638+0000 7f0912f59640 10 monclient: monmap: 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:31.640 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.639+0000 7f0912f59640 1 librados: starting msgr at 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.639+0000 7f0912f59640 1 librados: starting objecter 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.639+0000 7f0912f59640 1 Processor -- start 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.639+0000 7f0912f59640 1 -- start start 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.639+0000 7f0912f59640 1 librados: setting wanted keys 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.639+0000 7f0912f59640 1 librados: calling monclient init 2024-03-25T19:49:31.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.639+0000 7f0912f59640 10 monclient: init 2024-03-25T19:49:31.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 10 monclient(hunting): picked mon.noname-b con 0x7f090c0b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 10 monclient(hunting): picked mon.noname-a con 0x7f090c0b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 10 monclient(hunting): picked mon.noname-c con 0x7f090c14be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c137d90 con 0x7f090c0b4d60 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c142110 con 0x7f090c0b1730 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090c140b20 con 0x7f090c14be90 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.640+0000 7f0912f59640 10 monclient(hunting): authenticate will time out at 5346.904311s 2024-03-25T19:49:31.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.654+0000 7f090bfff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f090c0b4d60 0x7f090c133fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:50098/0 (socket says 172.21.15.50:50098) 2024-03-25T19:49:31.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.654+0000 7f090bfff640 1 -- 172.21.15.50:0/773529890 learned_addr learned my addr 172.21.15.50:0/773529890 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:31.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.655+0000 7f09097fa640 10 client.?.objecter ms_handle_connect 0x7f090c0b4d60 2024-03-25T19:49:31.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.655+0000 7f09097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.663+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818884319 0 0) 0x7f090c137d90 con 0x7f090c0b4d60 2024-03-25T19:49:31.662 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.663+0000 7f09097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.663+0000 7f09097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.663+0000 7f09097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.663+0000 7f09097fa640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:49:31.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.663+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0004eb0 con 0x7f090c0b4d60 2024-03-25T19:49:31.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.665+0000 7f09097fa640 10 client.?.objecter ms_handle_connect 0x7f090c14be90 2024-03-25T19:49:31.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.665+0000 7f09097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.666+0000 7f09097fa640 10 client.?.objecter ms_handle_connect 0x7f090c0b1730 2024-03-25T19:49:31.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.666+0000 7f09097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.670+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148289695 0 0) 0x7f090c140b20 con 0x7f090c14be90 2024-03-25T19:49:31.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.670+0000 7f09097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.670+0000 7f09097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.670+0000 7f09097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.670+0000 7f09097fa640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:49:31.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.670+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c137d90 con 0x7f090c14be90 2024-03-25T19:49:31.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.679+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809644207 0 0) 0x7f090c142110 con 0x7f090c0b1730 2024-03-25T19:49:31.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.679+0000 7f09097fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.679+0000 7f09097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.679+0000 7f09097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.679+0000 7f09097fa640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:49:31.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.679+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f090c140b20 con 0x7f090c0b1730 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.681+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682411661 0 0) 0x7f08f0004eb0 con 0x7f090c0b4d60 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.681+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f090c142110 con 0x7f090c0b4d60 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.681+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f08f8001bc0 con 0x7f090c0b4d60 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.681+0000 7f09097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.681+0000 7f09097fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.681+0000 7f09097fa640 10 monclient(hunting): dump: 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:31.681 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:31.682 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:31.682 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:31.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.681+0000 7f09097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.689+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545822715 0 0) 0x7f090c137d90 con 0x7f090c14be90 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.689+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08f0004eb0 con 0x7f090c14be90 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.689+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0904001b00 con 0x7f090c14be90 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.689+0000 7f09097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.689+0000 7f09097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.689+0000 7f09097fa640 10 monclient(hunting): dump: 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:31.689 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:31.690 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:31.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1256516975 0 0) 0x7f08f0004eb0 con 0x7f090c14be90 2024-03-25T19:49:31.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 >> v1:172.21.15.76:6789/0 conn(0x7f090c0b1730 legacy=0x7f090c140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 >> v1:172.21.15.50:6789/0 conn(0x7f090c0b4d60 legacy=0x7f090c133fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 1 monclient: found mon.c 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c140cb0 con 0x7f090c14be90 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f09097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 5 monclient: authenticate success, global_id 4604 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 10 monclient: _renew_subs 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090c140ef0 con 0x7f090c14be90 2024-03-25T19:49:31.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:31.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 10 monclient: _renew_subs 2024-03-25T19:49:31.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:31.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c142710 con 0x7f090c14be90 2024-03-25T19:49:31.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.700+0000 7f0912f59640 1 librados: init done 2024-03-25T19:49:31.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.701+0000 7f0912f59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.701+0000 7f0912f59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.701+0000 7f0912f59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:31.703 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.701+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c0b2270 con 0x7f090c14be90 2024-03-25T19:49:31.917 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.918+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0904001fb0 con 0x7f090c14be90 2024-03-25T19:49:31.918 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.918+0000 7f09097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:31.918 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:31.918+0000 7f090affd640 4 set_mon_vals no callback set 2024-03-25T19:49:32.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.017+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f0904003b10 con 0x7f090c14be90 2024-03-25T19:49:32.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.017+0000 7f09097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.017+0000 7f09097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.017+0000 7f09097fa640 10 monclient: dump: 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:32.018 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:32.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.019+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f0904003cf0 con 0x7f090c14be90 2024-03-25T19:49:32.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.019+0000 7f0910cf5640 10 monclient: get_auth_request con 0x7f08f006df70 auth_method 0 2024-03-25T19:49:32.019 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.019+0000 7f09097fa640 10 client.4604.objecter ms_handle_connect 0x7f08f006df70 2024-03-25T19:49:32.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f0904053700 con 0x7f090c14be90 2024-03-25T19:49:32.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 10 client.4604.objecter ms_dispatch 0x7f090c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:32.025 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:32.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:32.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T19:49:32.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f090404f210 con 0x7f090c14be90 2024-03-25T19:49:32.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:32.026 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.026+0000 7f09097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:32.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.283+0000 7f0912f59640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.283+0000 7f0912f59640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.283+0000 7f0912f59640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:49:32.283 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.283+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f090c0627b0 con 0x7f090c14be90 2024-03-25T19:49:32.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.321+0000 7f09097fa640 1 -- 172.21.15.50:0/773529890 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f0904051140 con 0x7f090c14be90 2024-03-25T19:49:32.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.321+0000 7f09097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.321+0000 7f09097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:32.321 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:32.321 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:32.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 10 librados: watch_flush enter 2024-03-25T19:49:32.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 10 librados: watch_flush exit 2024-03-25T19:49:32.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T19:49:32.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 10 client.4604.objecter successfully canceled tick 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 >> v1:172.21.15.76:6800/2572556811 conn(0x7f08f006df70 legacy=0x7f08f0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 10 monclient: shutdown 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 >> v1:172.21.15.50:6790/0 conn(0x7f090c14be90 legacy=0x7f090c1e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 shutdown_connections 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.323+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 >> 172.21.15.50:0/773529890 conn(0x7f090c0617a0 msgr2=0x7f090c13b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.324+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 shutdown_connections 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.324+0000 7f0912f59640 1 -- 172.21.15.50:0/773529890 wait complete. 2024-03-25T19:49:32.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:32.324+0000 7f0912f59640 1 librados: shutdown 2024-03-25T19:49:32.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:39.141 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:39.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.258+0000 7f1946ce4640 10 monclient: get_monmap_and_config 2024-03-25T19:49:39.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.258+0000 7f1946ce4640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.258+0000 7f1946ce4640 10 monclient: monmap: 2024-03-25T19:49:39.258 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:39.258 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:39.258 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:39.259607+0000 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:39.259607+0000 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:39.259 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:39.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.259+0000 7f1946ce4640 1 Processor -- start 2024-03-25T19:49:39.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.259+0000 7f1946ce4640 1 -- start start 2024-03-25T19:49:39.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.259+0000 7f1946ce4640 10 monclient: init 2024-03-25T19:49:39.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.260 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 10 monclient(hunting): picked mon.noname-a con 0x7f19400b10d0 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 10 monclient(hunting): picked mon.noname-c con 0x7f19400b4360 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 10 monclient(hunting): picked mon.noname-b con 0x7f19400a9470 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400a9ab0 con 0x7f19400b10d0 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400b2400 con 0x7f19400b4360 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400aa2a0 con 0x7f19400a9470 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1946ce4640 10 monclient(hunting): authenticate will time out at 5354.523861s 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1945281640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f19400a9470 0x7f194014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:49836/0 (socket says 172.21.15.50:49836) 2024-03-25T19:49:39.261 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.260+0000 7f1945281640 1 -- 172.21.15.50:0/1583209136 learned_addr learned my addr 172.21.15.50:0/1583209136 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:39.306 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.307+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022940474 0 0) 0x7f19400b2400 con 0x7f19400b4360 2024-03-25T19:49:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.307+0000 7f193f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.307+0000 7f193f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.307+0000 7f193f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.307+0000 7f193f7fe640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:49:39.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.307+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f192c003520 con 0x7f19400b4360 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.313+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802112225 0 0) 0x7f19400a9ab0 con 0x7f19400b10d0 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.313+0000 7f193f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.313+0000 7f193f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.313+0000 7f193f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.313+0000 7f193f7fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:49:39.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.313+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19400b2400 con 0x7f19400b10d0 2024-03-25T19:49:39.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3508567348 0 0) 0x7f192c003520 con 0x7f19400b4360 2024-03-25T19:49:39.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 >> v1:172.21.15.50:6790/0 conn(0x7f19400a9470 legacy=0x7f194014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.318 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 >> v1:172.21.15.50:6789/0 conn(0x7f19400b10d0 legacy=0x7f19400b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400b14b0 con 0x7f19400b4360 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.318+0000 7f193f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.319+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1930001690 con 0x7f19400b4360 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.319+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.319+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.319+0000 7f193f7fe640 10 monclient: dump: 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:39.319 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.319+0000 7f1946ce4640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:49:39.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.319+0000 7f1946ce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19300017f0 con 0x7f19400b4360 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f193f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f193f7fe640 1 -- 172.21.15.50:0/1583209136 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1930001c00 con 0x7f19400b4360 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f193f7fe640 10 monclient: dump: 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.333 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.334 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:39.334 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:39.334 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:39.334 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:39.334 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:39.334 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f1946ce4640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:39.334 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.333+0000 7f1946ce4640 4 set_mon_vals no callback set 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.334+0000 7f1946ce4640 10 monclient: shutdown 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.334+0000 7f1946ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.334+0000 7f1946ce4640 1 -- 172.21.15.50:0/1583209136 >> v1:172.21.15.76:6789/0 conn(0x7f19400b4360 legacy=0x7f19400b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.334+0000 7f1946ce4640 1 -- 172.21.15.50:0/1583209136 shutdown_connections 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.334+0000 7f1946ce4640 1 -- 172.21.15.50:0/1583209136 >> 172.21.15.50:0/1583209136 conn(0x7f19400617a0 msgr2=0x7f1940061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:39.336 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.334+0000 7f1946ce4640 1 -- 172.21.15.50:0/1583209136 shutdown_connections 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.334+0000 7f1946ce4640 1 -- 172.21.15.50:0/1583209136 wait complete. 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 10 monclient: monmap: 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:39.337 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 1 librados: starting msgr at 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 1 librados: starting objecter 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 1 Processor -- start 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 1 -- start start 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 1 librados: setting wanted keys 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 1 librados: calling monclient init 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.335+0000 7f1946ce4640 10 monclient: init 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 10 monclient(hunting): picked mon.noname-b con 0x7f19400a9470 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 10 monclient(hunting): picked mon.noname-c con 0x7f19400b10d0 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 10 monclient(hunting): picked mon.noname-a con 0x7f19400b4360 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194013d5c0 con 0x7f19400b4360 2024-03-25T19:49:39.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19401309c0 con 0x7f19400a9470 2024-03-25T19:49:39.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194013c010 con 0x7f19400b10d0 2024-03-25T19:49:39.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.339 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.336+0000 7f1946ce4640 10 monclient(hunting): authenticate will time out at 5354.600391s 2024-03-25T19:49:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.343+0000 7f1944a80640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f19400a9470 0x7f194013dfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:46236/0 (socket says 172.21.15.50:46236) 2024-03-25T19:49:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.343+0000 7f1944a80640 1 -- 172.21.15.50:0/1665949629 learned_addr learned my addr 172.21.15.50:0/1665949629 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.344+0000 7f193d7fa640 10 client.?.objecter ms_handle_connect 0x7f19400a9470 2024-03-25T19:49:39.343 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.344+0000 7f193d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.354+0000 7f193d7fa640 10 client.?.objecter ms_handle_connect 0x7f19400b4360 2024-03-25T19:49:39.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.354+0000 7f193d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.361+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279280857 0 0) 0x7f19401309c0 con 0x7f19400a9470 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.361+0000 7f193d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.360 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.361+0000 7f193d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.361+0000 7f193d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.361+0000 7f193d7fa640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:49:39.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.361+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1920004eb0 con 0x7f19400a9470 2024-03-25T19:49:39.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.365+0000 7f193d7fa640 10 client.?.objecter ms_handle_connect 0x7f19400b10d0 2024-03-25T19:49:39.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.365+0000 7f193d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.371+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832339968 0 0) 0x7f1920004eb0 con 0x7f19400a9470 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.371+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19401309c0 con 0x7f19400a9470 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.371+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1938001d90 con 0x7f19400a9470 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.371+0000 7f193d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.371+0000 7f193d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.371+0000 7f193d7fa640 10 monclient(hunting): dump: 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.371 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.372 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:39.372 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:39.372 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:39.372 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:39.372 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:39.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.371+0000 7f193d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:39.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.379+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461171764 0 0) 0x7f194013d5c0 con 0x7f19400b4360 2024-03-25T19:49:39.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.379+0000 7f193d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.379+0000 7f193d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.379+0000 7f193d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.379+0000 7f193d7fa640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:49:39.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.379+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1920004eb0 con 0x7f19400b4360 2024-03-25T19:49:39.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 979723724 0 0) 0x7f19401309c0 con 0x7f19400a9470 2024-03-25T19:49:39.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 >> v1:172.21.15.50:6790/0 conn(0x7f19400b10d0 legacy=0x7f194012fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 >> v1:172.21.15.50:6789/0 conn(0x7f19400b4360 legacy=0x7f19401302b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 1 monclient: found mon.b 2024-03-25T19:49:39.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194013d830 con 0x7f19400a9470 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f193d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 5 monclient: authenticate success, global_id 4612 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 10 monclient: _renew_subs 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194013db10 con 0x7f19400a9470 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 10 monclient: _renew_subs 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1940138d40 con 0x7f19400a9470 2024-03-25T19:49:39.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.383+0000 7f1946ce4640 1 librados: init done 2024-03-25T19:49:39.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.385+0000 7f1946ce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.385+0000 7f1946ce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.385+0000 7f1946ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:39.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.385+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19400627b0 con 0x7f19400a9470 2024-03-25T19:49:39.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.600+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1938002010 con 0x7f19400a9470 2024-03-25T19:49:39.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.600+0000 7f193d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.600+0000 7f193effd640 4 set_mon_vals no callback set 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.666+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f1938003de0 con 0x7f19400a9470 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.666+0000 7f193d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.666+0000 7f193d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.666+0000 7f193d7fa640 10 monclient: dump: 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.666 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.667+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f1938004000 con 0x7f19400a9470 2024-03-25T19:49:39.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.668+0000 7f193ffff640 10 monclient: get_auth_request con 0x7f192006dd10 auth_method 0 2024-03-25T19:49:39.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.668+0000 7f193d7fa640 10 client.4612.objecter ms_handle_connect 0x7f192006dd10 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.672+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f1938054d10 con 0x7f19400a9470 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.672+0000 7f193d7fa640 10 client.4612.objecter ms_dispatch 0x7f1940000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.672+0000 7f193d7fa640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.672+0000 7f193d7fa640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:39.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.672+0000 7f193d7fa640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T19:49:39.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.674+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1938050650 con 0x7f19400a9470 2024-03-25T19:49:39.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.674+0000 7f193d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.674+0000 7f193d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:39.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.931+0000 7f1946ce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:39.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.931+0000 7f1946ce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:39.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.931+0000 7f1946ce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:39.931 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.931+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1940061e60 con 0x7f19400a9470 2024-03-25T19:49:39.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.964+0000 7f193d7fa640 1 -- 172.21.15.50:0/1665949629 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f1938052750 con 0x7f19400a9470 2024-03-25T19:49:39.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.964+0000 7f193d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:39.964 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.964+0000 7f193d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:39.964 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:39.965 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:39.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.966+0000 7f1946ce4640 10 librados: watch_flush enter 2024-03-25T19:49:39.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.966+0000 7f1946ce4640 10 librados: watch_flush exit 2024-03-25T19:49:39.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.966+0000 7f1946ce4640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T19:49:39.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.966+0000 7f1946ce4640 10 client.4612.objecter successfully canceled tick 2024-03-25T19:49:39.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 >> v1:172.21.15.76:6800/2572556811 conn(0x7f192006dd10 legacy=0x7f19200701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 10 monclient: shutdown 2024-03-25T19:49:39.967 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:39.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 >> v1:172.21.15.76:6789/0 conn(0x7f19400a9470 legacy=0x7f194013dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 shutdown_connections 2024-03-25T19:49:39.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 >> 172.21.15.50:0/1665949629 conn(0x7f19400617a0 msgr2=0x7f19400b64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:39.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 shutdown_connections 2024-03-25T19:49:39.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 1 -- 172.21.15.50:0/1665949629 wait complete. 2024-03-25T19:49:39.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:39.967+0000 7f1946ce4640 1 librados: shutdown 2024-03-25T19:49:39.978 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:40.804 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:01.310 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.857 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:01.363 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.894 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:01.400 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.903 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:01.374 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.914 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:01.384 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.918 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:01.424 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.929 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:01.435 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.949 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:01.419 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:40.964 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:01.470 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:46.785 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:46.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.894+0000 7f527a263640 10 monclient: get_monmap_and_config 2024-03-25T19:49:46.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.894+0000 7f527a263640 10 monclient: build_initial_monmap 2024-03-25T19:49:46.894 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.894+0000 7f527a263640 10 monclient: monmap: 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:46.895573+0000 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:46.895573+0000 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:46.895 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:46.896 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.895+0000 7f527a263640 1 Processor -- start 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.895+0000 7f527a263640 1 -- start start 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.895+0000 7f527a263640 10 monclient: init 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 10 monclient(hunting): picked mon.noname-a con 0x7f52740b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 10 monclient(hunting): picked mon.noname-b con 0x7f52740b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 10 monclient(hunting): picked mon.noname-c con 0x7f527414be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740af940 con 0x7f52740b4d60 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740b0a70 con 0x7f527414be90 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52740b0d50 con 0x7f52740b1730 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f527a263640 10 monclient(hunting): authenticate will time out at 5362.159910s 2024-03-25T19:49:46.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f52737fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f52740b1730 0x7f52740b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:54544/0 (socket says 172.21.15.50:54544) 2024-03-25T19:49:46.898 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.896+0000 7f52737fe640 1 -- 172.21.15.50:0/2826734880 learned_addr learned my addr 172.21.15.50:0/2826734880 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.948+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557129606 0 0) 0x7f52740af940 con 0x7f52740b4d60 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.948+0000 7f5272ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.948+0000 7f5272ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.948+0000 7f5272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.948+0000 7f5272ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:49:46.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.948+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f525c003520 con 0x7f52740b4d60 2024-03-25T19:49:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515672863 0 0) 0x7f525c003520 con 0x7f52740b4d60 2024-03-25T19:49:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 >> v1:172.21.15.50:6790/0 conn(0x7f52740b1730 legacy=0x7f52740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 >> v1:172.21.15.76:6789/0 conn(0x7f527414be90 legacy=0x7f527414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 1 monclient: found mon.noname-a 2024-03-25T19:49:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:46.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52740b1030 con 0x7f52740b4d60 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5264001690 con 0x7f52740b4d60 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f527a263640 5 monclient: authenticate success, global_id 4638 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f527a263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f5272ffd640 10 monclient: dump: 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:46.956 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:46.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.955+0000 7f527a263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52640017f0 con 0x7f52740b4d60 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f5272ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f5272ffd640 1 -- 172.21.15.50:0/2826734880 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5264001c00 con 0x7f52740b4d60 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f5272ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f5272ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f5272ffd640 10 monclient: dump: 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:46.969 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:46.970 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:46.970 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:46.970 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:46.970 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:46.970 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:46.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f527a263640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:46.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.969+0000 7f527a263640 4 set_mon_vals no callback set 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 10 monclient: shutdown 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 1 -- 172.21.15.50:0/2826734880 >> v1:172.21.15.50:6789/0 conn(0x7f52740b4d60 legacy=0x7f527413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:46.972 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 1 -- 172.21.15.50:0/2826734880 shutdown_connections 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 1 -- 172.21.15.50:0/2826734880 >> 172.21.15.50:0/2826734880 conn(0x7f52740617a0 msgr2=0x7f5274061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 1 -- 172.21.15.50:0/2826734880 shutdown_connections 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 1 -- 172.21.15.50:0/2826734880 wait complete. 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.970+0000 7f527a263640 10 monclient: build_initial_monmap 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 10 monclient: monmap: 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:46.973 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 1 librados: starting msgr at 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 1 librados: starting objecter 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 1 Processor -- start 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 1 -- start start 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 1 librados: setting wanted keys 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 1 librados: calling monclient init 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.971+0000 7f527a263640 10 monclient: init 2024-03-25T19:49:46.974 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 10 monclient(hunting): picked mon.noname-c con 0x7f52740b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 10 monclient(hunting): picked mon.noname-a con 0x7f52740b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 10 monclient(hunting): picked mon.noname-b con 0x7f527414be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527413a940 con 0x7f52740b4d60 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527413ab90 con 0x7f527414be90 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52741d7840 con 0x7f52740b1730 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:46.975 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.972+0000 7f527a263640 10 monclient(hunting): authenticate will time out at 5362.236320s 2024-03-25T19:49:46.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.988+0000 7f52737fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f52740b4d60 0x7f52741d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48104/0 (socket says 172.21.15.50:48104) 2024-03-25T19:49:46.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.988+0000 7f52737fe640 1 -- 172.21.15.50:0/3067162327 learned_addr learned my addr 172.21.15.50:0/3067162327 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:46.991 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.992+0000 7f5270ff9640 10 client.?.objecter ms_handle_connect 0x7f52740b4d60 2024-03-25T19:49:46.992 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.992+0000 7f5270ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:46.996 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.997+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452339758 0 0) 0x7f527413a940 con 0x7f52740b4d60 2024-03-25T19:49:46.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.997+0000 7f5270ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:46.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.997+0000 7f5270ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:46.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.997+0000 7f5270ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:46.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.997+0000 7f5270ff9640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:49:46.997 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:46.997+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260004eb0 con 0x7f52740b4d60 2024-03-25T19:49:47.009 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.009+0000 7f5270ff9640 10 client.?.objecter ms_handle_connect 0x7f527414be90 2024-03-25T19:49:47.009 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.009+0000 7f5270ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.016+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1933200875 0 0) 0x7f5260004eb0 con 0x7f52740b4d60 2024-03-25T19:49:47.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.016+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f527413a940 con 0x7f52740b4d60 2024-03-25T19:49:47.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.016+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f52680018b0 con 0x7f52740b4d60 2024-03-25T19:49:47.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.016+0000 7f5270ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.016+0000 7f5270ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:47.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.016+0000 7f5270ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:47.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.016+0000 7f5270ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:47.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.021+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791597620 0 0) 0x7f527413ab90 con 0x7f527414be90 2024-03-25T19:49:47.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.021+0000 7f5270ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.021+0000 7f5270ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.021+0000 7f5270ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.021+0000 7f5270ff9640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:49:47.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.021+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5260004eb0 con 0x7f527414be90 2024-03-25T19:49:47.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.025+0000 7f5270ff9640 10 client.?.objecter ms_handle_connect 0x7f52740b1730 2024-03-25T19:49:47.024 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.025+0000 7f5270ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.032+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800555912 0 0) 0x7f52741d7840 con 0x7f52740b1730 2024-03-25T19:49:47.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.032+0000 7f5270ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.032+0000 7f5270ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.032+0000 7f5270ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.032+0000 7f5270ff9640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:49:47.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.032+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f527413ab90 con 0x7f52740b1730 2024-03-25T19:49:47.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4249452123 0 0) 0x7f527413a940 con 0x7f52740b4d60 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 >> v1:172.21.15.50:6790/0 conn(0x7f52740b1730 legacy=0x7f527413bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 >> v1:172.21.15.76:6789/0 conn(0x7f527414be90 legacy=0x7f52741e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 1 monclient: found mon.a 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527413c240 con 0x7f52740b4d60 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f5270ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 5 monclient: authenticate success, global_id 4641 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 10 monclient: _renew_subs 2024-03-25T19:49:47.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5274139ce0 con 0x7f52740b4d60 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 10 client.4641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 10 monclient: _renew_subs 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52741d8650 con 0x7f52740b4d60 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.036+0000 7f527a263640 1 librados: init done 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.038+0000 7f527a263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.038+0000 7f527a263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.038+0000 7f527a263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:49:47.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.038+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52741401f0 con 0x7f52740b4d60 2024-03-25T19:49:47.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.378+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52680047e0 con 0x7f52740b4d60 2024-03-25T19:49:47.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.378+0000 7f5270ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:47.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.378+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5268004bf0 con 0x7f52740b4d60 2024-03-25T19:49:47.378 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.378+0000 7f5270ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.378+0000 7f5270ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.378+0000 7f5270ff9640 10 monclient: dump: 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:47.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.378+0000 7f52727fc640 4 set_mon_vals no callback set 2024-03-25T19:49:47.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.379+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f5268004e10 con 0x7f52740b4d60 2024-03-25T19:49:47.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.380+0000 7f5273fff640 10 monclient: get_auth_request con 0x7f526006de10 auth_method 0 2024-03-25T19:49:47.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.380+0000 7f5270ff9640 10 client.4641.objecter ms_handle_connect 0x7f526006de10 2024-03-25T19:49:47.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f52680548a0 con 0x7f52740b4d60 2024-03-25T19:49:47.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 10 client.4641.objecter ms_dispatch 0x7f5274000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:47.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 3 client.4641.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:47.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 3 client.4641.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:47.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 20 client.4641.objecter dump_active .. 0 homeless 2024-03-25T19:49:47.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52680503b0 con 0x7f52740b4d60 2024-03-25T19:49:47.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.385+0000 7f5270ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:47.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.641+0000 7f527a263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.641+0000 7f527a263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.641+0000 7f527a263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:49:47.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.641+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52740627b0 con 0x7f52740b4d60 2024-03-25T19:49:47.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.673+0000 7f5270ff9640 1 -- 172.21.15.50:0/3067162327 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f52680522e0 con 0x7f52740b4d60 2024-03-25T19:49:47.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.673+0000 7f5270ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.673+0000 7f5270ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:47.673 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:47.673 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.675+0000 7f527a263640 10 librados: watch_flush enter 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.675+0000 7f527a263640 10 librados: watch_flush exit 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.675+0000 7f527a263640 20 client.4641.objecter shutdown clearing up homeless session... 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.675+0000 7f527a263640 10 client.4641.objecter successfully canceled tick 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 >> v1:172.21.15.76:6800/2572556811 conn(0x7f526006de10 legacy=0x7f52600702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 10 monclient: shutdown 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 >> v1:172.21.15.50:6789/0 conn(0x7f52740b4d60 legacy=0x7f52741d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 shutdown_connections 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 >> 172.21.15.50:0/3067162327 conn(0x7f52740617a0 msgr2=0x7f5274137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:47.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 shutdown_connections 2024-03-25T19:49:47.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 1 -- 172.21.15.50:0/3067162327 wait complete. 2024-03-25T19:49:47.677 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:47.676+0000 7f527a263640 1 librados: shutdown 2024-03-25T19:49:47.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:54.496 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.611+0000 7f18b38dd640 10 monclient: get_monmap_and_config 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.611+0000 7f18b38dd640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.611+0000 7f18b38dd640 10 monclient: monmap: 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:49:54.613237+0000 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:49:54.613237+0000 2024-03-25T19:49:54.612 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.613 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:54.613 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:54.613 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:49:54.613 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:49:54.613 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:54.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.612+0000 7f18b38dd640 1 Processor -- start 2024-03-25T19:49:54.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.612+0000 7f18b38dd640 1 -- start start 2024-03-25T19:49:54.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.612+0000 7f18b38dd640 10 monclient: init 2024-03-25T19:49:54.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac142520 con 0x7f18ac1430c0 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0b28a0 con 0x7f18ac13f710 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0b2a30 con 0x7f18ac0a98f0 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.615 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.613+0000 7f18b38dd640 10 monclient(hunting): authenticate will time out at 5369.878317s 2024-03-25T19:49:54.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.619+0000 7f18b1679640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f18ac0a98f0 0x7f18ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:52146/0 (socket says 172.21.15.50:52146) 2024-03-25T19:49:54.620 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.619+0000 7f18b1679640 1 -- 172.21.15.50:0/3668775168 learned_addr learned my addr 172.21.15.50:0/3668775168 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:54.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.627+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746230354 0 0) 0x7f18ac0b2a30 con 0x7f18ac0a98f0 2024-03-25T19:49:54.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.627+0000 7f18a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.627+0000 7f18a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.627+0000 7f18a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.627+0000 7f18a3fff640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:49:54.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.627+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898003520 con 0x7f18ac0a98f0 2024-03-25T19:49:54.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.633+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608772934 0 0) 0x7f18ac0b28a0 con 0x7f18ac13f710 2024-03-25T19:49:54.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.633+0000 7f18a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.633+0000 7f18a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.633+0000 7f18a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.633+0000 7f18a3fff640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:49:54.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.633+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ac0b2a30 con 0x7f18ac13f710 2024-03-25T19:49:54.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.643+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177421315 0 0) 0x7f18ac142520 con 0x7f18ac1430c0 2024-03-25T19:49:54.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.643+0000 7f18a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.643+0000 7f18a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.643+0000 7f18a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.643+0000 7f18a3fff640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:49:54.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.643+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18ac0b28a0 con 0x7f18ac1430c0 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776512998 0 0) 0x7f18ac0b28a0 con 0x7f18ac1430c0 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 >> v1:172.21.15.50:6790/0 conn(0x7f18ac0a98f0 legacy=0x7f18ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 >> v1:172.21.15.76:6789/0 conn(0x7f18ac13f710 legacy=0x7f18ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 1 monclient: found mon.noname-a 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac0b44c0 con 0x7f18ac1430c0 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:54.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f18a4001930 con 0x7f18ac1430c0 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18a3fff640 10 monclient: dump: 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:54.656 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:54.657 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:54.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18b38dd640 5 monclient: authenticate success, global_id 4644 2024-03-25T19:49:54.657 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.653+0000 7f18b38dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18a4001a90 con 0x7f18ac1430c0 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18a3fff640 1 -- 172.21.15.50:0/3668775168 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f18a4001ee0 con 0x7f18ac1430c0 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18b38dd640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18b38dd640 4 set_mon_vals no callback set 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.664+0000 7f18a3fff640 10 monclient: dump: 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:54.665 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:54.666 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.665+0000 7f18b38dd640 10 monclient: shutdown 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.665+0000 7f18b38dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.665+0000 7f18b38dd640 1 -- 172.21.15.50:0/3668775168 >> v1:172.21.15.50:6789/0 conn(0x7f18ac1430c0 legacy=0x7f18ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.665+0000 7f18b38dd640 1 -- 172.21.15.50:0/3668775168 shutdown_connections 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.665+0000 7f18b38dd640 1 -- 172.21.15.50:0/3668775168 >> 172.21.15.50:0/3668775168 conn(0x7f18ac0b7ea0 msgr2=0x7f18ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.665+0000 7f18b38dd640 1 -- 172.21.15.50:0/3668775168 shutdown_connections 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.665+0000 7f18b38dd640 1 -- 172.21.15.50:0/3668775168 wait complete. 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 10 monclient: monmap: 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 1 librados: starting msgr at 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 1 librados: starting objecter 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 1 Processor -- start 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 1 -- start start 2024-03-25T19:49:54.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 1 librados: setting wanted keys 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 1 librados: calling monclient init 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.666+0000 7f18b38dd640 10 monclient: init 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac0a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac13f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac1430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac139990 con 0x7f18ac0a98f0 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac1d51f0 con 0x7f18ac1430c0 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac137a00 con 0x7f18ac13f710 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.667+0000 7f18b38dd640 10 monclient(hunting): authenticate will time out at 5369.932004s 2024-03-25T19:49:54.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.673+0000 7f18b1e7a640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f18ac1430c0 0x7f18ac1db410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:50766/0 (socket says 172.21.15.50:50766) 2024-03-25T19:49:54.673 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.673+0000 7f18b1e7a640 1 -- 172.21.15.50:0/3459512828 learned_addr learned my addr 172.21.15.50:0/3459512828 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:49:54.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.675+0000 7f18a1ffb640 10 client.?.objecter ms_handle_connect 0x7f18ac1430c0 2024-03-25T19:49:54.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.675+0000 7f18a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.678+0000 7f18a1ffb640 10 client.?.objecter ms_handle_connect 0x7f18ac13f710 2024-03-25T19:49:54.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.678+0000 7f18a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.678+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198419905 0 0) 0x7f18ac1d51f0 con 0x7f18ac1430c0 2024-03-25T19:49:54.679 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.679+0000 7f18a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.679+0000 7f18a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.679+0000 7f18a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.679+0000 7f18a1ffb640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:49:54.680 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.679+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1890004eb0 con 0x7f18ac1430c0 2024-03-25T19:49:54.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.684+0000 7f18a1ffb640 10 client.?.objecter ms_handle_connect 0x7f18ac0a98f0 2024-03-25T19:49:54.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.684+0000 7f18a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.686+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778368288 0 0) 0x7f1890004eb0 con 0x7f18ac1430c0 2024-03-25T19:49:54.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.686+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18ac1d51f0 con 0x7f18ac1430c0 2024-03-25T19:49:54.686 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.686+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f18a4001ea0 con 0x7f18ac1430c0 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.686+0000 7f18a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.686+0000 7f18a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.686+0000 7f18a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:54.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.686+0000 7f18a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:54.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3392485614 0 0) 0x7f18ac1d51f0 con 0x7f18ac1430c0 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 >> v1:172.21.15.50:6790/0 conn(0x7f18ac13f710 legacy=0x7f18ac1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 >> v1:172.21.15.50:6789/0 conn(0x7f18ac0a98f0 legacy=0x7f18ac137f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 1 monclient: found mon.b 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.692+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac138630 con 0x7f18ac1430c0 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18a1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 5 monclient: authenticate success, global_id 4624 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 10 monclient: _renew_subs 2024-03-25T19:49:54.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:54.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac1e2dd0 con 0x7f18ac1430c0 2024-03-25T19:49:54.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:54.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 10 monclient: _renew_subs 2024-03-25T19:49:54.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:54.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac1e3140 con 0x7f18ac1430c0 2024-03-25T19:49:54.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.693+0000 7f18b38dd640 1 librados: init done 2024-03-25T19:49:54.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.694+0000 7f18b38dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.694+0000 7f18b38dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.694+0000 7f18b38dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:54.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.694+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac0627b0 con 0x7f18ac1430c0 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.978+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18a4002000 con 0x7f18ac1430c0 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.978+0000 7f18a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.978+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f18a4003dc0 con 0x7f18ac1430c0 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.978+0000 7f18a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.978+0000 7f18a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:54.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.978+0000 7f18a1ffb640 10 monclient: dump: 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:49:54.980 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.978+0000 7f18a37fe640 4 set_mon_vals no callback set 2024-03-25T19:49:54.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.979+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f18a4003fe0 con 0x7f18ac1430c0 2024-03-25T19:49:54.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.980+0000 7f18b1679640 10 monclient: get_auth_request con 0x7f189006dd00 auth_method 0 2024-03-25T19:49:54.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.980+0000 7f18a1ffb640 10 client.4624.objecter ms_handle_connect 0x7f189006dd00 2024-03-25T19:49:54.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.984+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f18a4054b20 con 0x7f18ac1430c0 2024-03-25T19:49:54.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.984+0000 7f18a1ffb640 10 client.4624.objecter ms_dispatch 0x7f18ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:54.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.984+0000 7f18a1ffb640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:54.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.984+0000 7f18a1ffb640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:54.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.984+0000 7f18a1ffb640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T19:49:54.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.985+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18a4050630 con 0x7f18ac1430c0 2024-03-25T19:49:54.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.985+0000 7f18a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.986 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:54.985+0000 7f18a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:55.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.243+0000 7f18b38dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.243+0000 7f18b38dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.243+0000 7f18b38dd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:49:55.246 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.243+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18ac13e100 con 0x7f18ac1430c0 2024-03-25T19:49:55.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.426+0000 7f18a1ffb640 1 -- 172.21.15.50:0/3459512828 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f18a4052560 con 0x7f18ac1430c0 2024-03-25T19:49:55.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.426+0000 7f18a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.426+0000 7f18a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:55.427 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:49:55.427 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.428+0000 7f18b38dd640 10 librados: watch_flush enter 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.428+0000 7f18b38dd640 10 librados: watch_flush exit 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.428+0000 7f18b38dd640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 10 client.4624.objecter successfully canceled tick 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 >> v1:172.21.15.76:6800/2572556811 conn(0x7f189006dd00 legacy=0x7f18900701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 10 monclient: shutdown 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 >> v1:172.21.15.76:6789/0 conn(0x7f18ac1430c0 legacy=0x7f18ac1db410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 shutdown_connections 2024-03-25T19:49:55.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 >> 172.21.15.50:0/3459512828 conn(0x7f18ac0b7ea0 msgr2=0x7f18ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:55.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 shutdown_connections 2024-03-25T19:49:55.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 1 -- 172.21.15.50:0/3459512828 wait complete. 2024-03-25T19:49:55.431 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:49:55.429+0000 7f18b38dd640 1 librados: shutdown 2024-03-25T19:49:55.442 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:00.854 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:21.359 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:00.916 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:21.422 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:00.950 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:21.455 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:00.986 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:21.457 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:01.004 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:21.475 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:01.006 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:21.512 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:01.016 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:21.521 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:01.027 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:21.498 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:01.036 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:21.542 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:02.248 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.353+0000 7fe00fd3b640 10 monclient: get_monmap_and_config 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.353+0000 7fe00fd3b640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.353+0000 7fe00fd3b640 10 monclient: monmap: 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:02.354122+0000 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:02.354122+0000 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:02.353 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.353+0000 7fe00fd3b640 1 Processor -- start 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 1 -- start start 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient: init 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0080a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient(hunting): picked mon.noname-b con 0x7fe00813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0081430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe008142520 con 0x7fe0080a98f0 2024-03-25T19:50:02.355 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0080b28a0 con 0x7fe0081430c0 2024-03-25T19:50:02.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0080b2a30 con 0x7fe00813f710 2024-03-25T19:50:02.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.354+0000 7fe00fd3b640 10 monclient(hunting): authenticate will time out at 5377.618430s 2024-03-25T19:50:02.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.355+0000 7fe00e2d8640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe0081430c0 0x7fe00814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:56266/0 (socket says 172.21.15.50:56266) 2024-03-25T19:50:02.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.355+0000 7fe00e2d8640 1 -- 172.21.15.50:0/1613673184 learned_addr learned my addr 172.21.15.50:0/1613673184 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:02.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.421+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533669010 0 0) 0x7fe008142520 con 0x7fe0080a98f0 2024-03-25T19:50:02.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.421+0000 7fe00cad5640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.421+0000 7fe00cad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.421+0000 7fe00cad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.421+0000 7fe00cad5640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:50:02.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.421+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff4003520 con 0x7fe0080a98f0 2024-03-25T19:50:02.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411094031 0 0) 0x7fdff4003520 con 0x7fe0080a98f0 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 >> v1:172.21.15.50:6790/0 conn(0x7fe00813f710 legacy=0x7fe0081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 >> v1:172.21.15.76:6789/0 conn(0x7fe0081430c0 legacy=0x7fe00814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 1 monclient: found mon.noname-a 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0080b44c0 con 0x7fe0080a98f0 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00fd3b640 5 monclient: authenticate success, global_id 4650 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe004001690 con 0x7fe0080a98f0 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.432+0000 7fe00fd3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:02.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.433+0000 7fe00cad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.433+0000 7fe00cad5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.433+0000 7fe00cad5640 10 monclient: dump: 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:02.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.433+0000 7fe00fd3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0040017f0 con 0x7fe0080a98f0 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00cad5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00cad5640 1 -- 172.21.15.50:0/1613673184 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe004001c00 con 0x7fe0080a98f0 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00fd3b640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00fd3b640 4 set_mon_vals no callback set 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00cad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00cad5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00cad5640 10 monclient: dump: 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:02.462 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:02.463 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:02.463 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00fd3b640 10 monclient: shutdown 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00fd3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.462+0000 7fe00fd3b640 1 -- 172.21.15.50:0/1613673184 >> v1:172.21.15.50:6789/0 conn(0x7fe0080a98f0 legacy=0x7fe0080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.463+0000 7fe00fd3b640 1 -- 172.21.15.50:0/1613673184 shutdown_connections 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.463+0000 7fe00fd3b640 1 -- 172.21.15.50:0/1613673184 >> 172.21.15.50:0/1613673184 conn(0x7fe0080b7ea0 msgr2=0x7fe0080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.463+0000 7fe00fd3b640 1 -- 172.21.15.50:0/1613673184 shutdown_connections 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.463+0000 7fe00fd3b640 1 -- 172.21.15.50:0/1613673184 wait complete. 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 10 monclient: monmap: 2024-03-25T19:50:02.465 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 1 librados: starting msgr at 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 1 librados: starting objecter 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 1 Processor -- start 2024-03-25T19:50:02.466 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 1 -- start start 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 1 librados: setting wanted keys 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 1 librados: calling monclient init 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.464+0000 7fe00fd3b640 10 monclient: init 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 10 monclient(hunting): picked mon.noname-b con 0x7fe0080a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 10 monclient(hunting): picked mon.noname-c con 0x7fe00813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0081430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0081d82d0 con 0x7fe0081430c0 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0080abb90 con 0x7fe0080a98f0 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0081d76a0 con 0x7fe00813f710 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.467 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.465+0000 7fe00fd3b640 10 monclient(hunting): authenticate will time out at 5377.729116s 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.475+0000 7fe00d2d6640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fe00813f710 0x7fe0081d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:38342/0 (socket says 172.21.15.50:38342) 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.475+0000 7fe00d2d6640 1 -- 172.21.15.50:0/3870080344 learned_addr learned my addr 172.21.15.50:0/3870080344 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:02.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.476+0000 7fdffa7fc640 10 client.?.objecter ms_handle_connect 0x7fe00813f710 2024-03-25T19:50:02.476 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.476+0000 7fdffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.480 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.480+0000 7fdffa7fc640 10 client.?.objecter ms_handle_connect 0x7fe0080a98f0 2024-03-25T19:50:02.480 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.480+0000 7fdffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.483+0000 7fdffa7fc640 10 client.?.objecter ms_handle_connect 0x7fe0081430c0 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.483+0000 7fdffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.487 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.488+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051176946 0 0) 0x7fe0080abb90 con 0x7fe0080a98f0 2024-03-25T19:50:02.487 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.488+0000 7fdffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.488+0000 7fdffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.488+0000 7fdffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.488+0000 7fdffa7fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:50:02.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.488+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfec004eb0 con 0x7fe0080a98f0 2024-03-25T19:50:02.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.492+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329765698 0 0) 0x7fe0081d82d0 con 0x7fe0081430c0 2024-03-25T19:50:02.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.492+0000 7fdffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.492+0000 7fdffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.491 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.492+0000 7fdffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.492 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.492+0000 7fdffa7fc640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:50:02.492 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.492+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0080abb90 con 0x7fe0081430c0 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.494+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1554937275 0 0) 0x7fdfec004eb0 con 0x7fe0080a98f0 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.494+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0081d82d0 con 0x7fe0080a98f0 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.494+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe004001c00 con 0x7fe0080a98f0 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.494+0000 7fdffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.494+0000 7fdffa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.494+0000 7fdffa7fc640 10 monclient(hunting): dump: 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.494 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.495 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:02.495 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:02.495 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:02.495 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:02.495 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:02.495 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.494+0000 7fdffa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:02.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.496+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865706888 0 0) 0x7fe0081d76a0 con 0x7fe00813f710 2024-03-25T19:50:02.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.496+0000 7fdffa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.496+0000 7fdffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.496+0000 7fdffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.496+0000 7fdffa7fc640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:50:02.496 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.496+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfec004eb0 con 0x7fe00813f710 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2955451154 0 0) 0x7fe0081d82d0 con 0x7fe0080a98f0 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 >> v1:172.21.15.50:6790/0 conn(0x7fe00813f710 legacy=0x7fe0081d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 >> v1:172.21.15.50:6789/0 conn(0x7fe0081430c0 legacy=0x7fe008134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 1 monclient: found mon.b 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe008134720 con 0x7fe0080a98f0 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fdffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.512 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 10 monclient: _renew_subs 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe008134970 con 0x7fe0080a98f0 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 10 monclient: _renew_subs 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe008137a10 con 0x7fe0080a98f0 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.512+0000 7fe00fd3b640 1 librados: init done 2024-03-25T19:50:02.513 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.513+0000 7fe00fd3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.514 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.513+0000 7fe00fd3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.514 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.513+0000 7fe00fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:02.514 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.513+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0081d7830 con 0x7fe0080a98f0 2024-03-25T19:50:02.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.838+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0040038a0 con 0x7fe0080a98f0 2024-03-25T19:50:02.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.838+0000 7fdffa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.838+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fe004003c70 con 0x7fe0080a98f0 2024-03-25T19:50:02.838 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.838+0000 7fdffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.838+0000 7fdffa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.838+0000 7fdffa7fc640 10 monclient: dump: 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:02.839 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.838+0000 7fdffbfff640 4 set_mon_vals no callback set 2024-03-25T19:50:02.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.839+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fe004003e90 con 0x7fe0080a98f0 2024-03-25T19:50:02.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.840+0000 7fe00d2d6640 10 monclient: get_auth_request con 0x7fdfec06de00 auth_method 0 2024-03-25T19:50:02.840 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.840+0000 7fdffa7fc640 10 client.4630.objecter ms_handle_connect 0x7fdfec06de00 2024-03-25T19:50:02.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.844+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fe004053760 con 0x7fe0080a98f0 2024-03-25T19:50:02.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.844+0000 7fdffa7fc640 10 client.4630.objecter ms_dispatch 0x7fe008000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:02.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.844+0000 7fdffa7fc640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:02.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.844+0000 7fdffa7fc640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:02.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.844+0000 7fdffa7fc640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T19:50:02.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.845+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe00404f270 con 0x7fe0080a98f0 2024-03-25T19:50:02.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.845+0000 7fdffa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:02.845+0000 7fdffa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:03.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.096+0000 7fe00fd3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.096+0000 7fe00fd3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.096+0000 7fe00fd3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:03.096 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.096+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0081d7490 con 0x7fe0080a98f0 2024-03-25T19:50:03.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.141+0000 7fdffa7fc640 1 -- 172.21.15.50:0/3870080344 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fe0040511a0 con 0x7fe0080a98f0 2024-03-25T19:50:03.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.141+0000 7fdffa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.141+0000 7fdffa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:03.141 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:03.141 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:03.144 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 10 librados: watch_flush enter 2024-03-25T19:50:03.144 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 10 librados: watch_flush exit 2024-03-25T19:50:03.144 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 10 client.4630.objecter successfully canceled tick 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 >> v1:172.21.15.76:6800/2572556811 conn(0x7fdfec06de00 legacy=0x7fdfec0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 10 monclient: shutdown 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 >> v1:172.21.15.76:6789/0 conn(0x7fe0080a98f0 legacy=0x7fe0081385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 shutdown_connections 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.144+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 >> 172.21.15.50:0/3870080344 conn(0x7fe0080b7ea0 msgr2=0x7fe008074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.145+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 shutdown_connections 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.145+0000 7fe00fd3b640 1 -- 172.21.15.50:0/3870080344 wait complete. 2024-03-25T19:50:03.145 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:03.145+0000 7fe00fd3b640 1 librados: shutdown 2024-03-25T19:50:03.156 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:09.963 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:10.068 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.068+0000 7fed75e65640 10 monclient: get_monmap_and_config 2024-03-25T19:50:10.068 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.068+0000 7fed75e65640 10 monclient: build_initial_monmap 2024-03-25T19:50:10.068 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.068+0000 7fed75e65640 10 monclient: monmap: 2024-03-25T19:50:10.068 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:10.069606+0000 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:10.069606+0000 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:10.069 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.069+0000 7fed75e65640 1 Processor -- start 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.069+0000 7fed75e65640 1 -- start start 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.069+0000 7fed75e65640 10 monclient: init 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 10 monclient(hunting): picked mon.noname-c con 0x7fed700b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 10 monclient(hunting): picked mon.noname-a con 0x7fed700b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 10 monclient(hunting): picked mon.noname-b con 0x7fed7014be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed700af940 con 0x7fed700b1730 2024-03-25T19:50:10.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed700b0a70 con 0x7fed700b4d60 2024-03-25T19:50:10.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed700b0d50 con 0x7fed7014be90 2024-03-25T19:50:10.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:10.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.070+0000 7fed75e65640 10 monclient(hunting): authenticate will time out at 5385.334645s 2024-03-25T19:50:10.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.075+0000 7fed6f7fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fed700b4d60 0x7fed7013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:45572/0 (socket says 172.21.15.50:45572) 2024-03-25T19:50:10.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.075+0000 7fed6f7fe640 1 -- 172.21.15.50:0/4057977093 learned_addr learned my addr 172.21.15.50:0/4057977093 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.084+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386753618 0 0) 0x7fed700b0a70 con 0x7fed700b4d60 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.084+0000 7fed6e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.084+0000 7fed6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.084+0000 7fed6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.084+0000 7fed6e7fc640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:50:10.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.084+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed58003520 con 0x7fed700b4d60 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547148976 0 0) 0x7fed58003520 con 0x7fed700b4d60 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 >> v1:172.21.15.50:6790/0 conn(0x7fed7014be90 legacy=0x7fed7014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 >> v1:172.21.15.50:6789/0 conn(0x7fed700b1730 legacy=0x7fed700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed700b1030 con 0x7fed700b4d60 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed60001690 con 0x7fed700b4d60 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed75e65640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed75e65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed6e7fc640 10 monclient: dump: 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.090 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:10.091 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:10.091 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:10.091 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:10.091 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:10.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.089+0000 7fed75e65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:10.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed600017f0 con 0x7fed700b4d60 2024-03-25T19:50:10.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed6e7fc640 1 -- 172.21.15.50:0/4057977093 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed60001c00 con 0x7fed700b4d60 2024-03-25T19:50:10.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed6e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:10.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed6e7fc640 10 monclient: dump: 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed75e65640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:10.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.093+0000 7fed75e65640 4 set_mon_vals no callback set 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.094+0000 7fed75e65640 10 monclient: shutdown 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.094+0000 7fed75e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.094+0000 7fed75e65640 1 -- 172.21.15.50:0/4057977093 >> v1:172.21.15.76:6789/0 conn(0x7fed700b4d60 legacy=0x7fed7013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.094+0000 7fed75e65640 1 -- 172.21.15.50:0/4057977093 shutdown_connections 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.094+0000 7fed75e65640 1 -- 172.21.15.50:0/4057977093 >> 172.21.15.50:0/4057977093 conn(0x7fed700617a0 msgr2=0x7fed70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.094+0000 7fed75e65640 1 -- 172.21.15.50:0/4057977093 shutdown_connections 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.094+0000 7fed75e65640 1 -- 172.21.15.50:0/4057977093 wait complete. 2024-03-25T19:50:10.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.095+0000 7fed75e65640 10 monclient: build_initial_monmap 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.095+0000 7fed75e65640 10 monclient: monmap: 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.095+0000 7fed75e65640 1 librados: starting msgr at 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.095+0000 7fed75e65640 1 librados: starting objecter 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.095+0000 7fed75e65640 1 Processor -- start 2024-03-25T19:50:10.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 1 -- start start 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 1 librados: setting wanted keys 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 1 librados: calling monclient init 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 10 monclient: init 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 10 monclient(hunting): picked mon.noname-c con 0x7fed700b1730 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 10 monclient(hunting): picked mon.noname-a con 0x7fed700b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.096+0000 7fed75e65640 10 monclient(hunting): picked mon.noname-b con 0x7fed7014be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.097+0000 7fed75e65640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed70141f40 con 0x7fed700b4d60 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.097+0000 7fed75e65640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed70142360 con 0x7fed7014be90 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.097+0000 7fed75e65640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed701424f0 con 0x7fed700b1730 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.097+0000 7fed75e65640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:10.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.097+0000 7fed75e65640 10 monclient(hunting): authenticate will time out at 5385.360838s 2024-03-25T19:50:10.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.103+0000 7fed6ffff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fed7014be90 0x7fed701e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:45600/0 (socket says 172.21.15.50:45600) 2024-03-25T19:50:10.102 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.103+0000 7fed6ffff640 1 -- 172.21.15.50:0/3878843441 learned_addr learned my addr 172.21.15.50:0/3878843441 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:10.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.104+0000 7fed57fff640 10 client.?.objecter ms_handle_connect 0x7fed7014be90 2024-03-25T19:50:10.103 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.104+0000 7fed57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.112+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428809466 0 0) 0x7fed70142360 con 0x7fed7014be90 2024-03-25T19:50:10.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.112+0000 7fed57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.112+0000 7fed57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.112+0000 7fed57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.112+0000 7fed57fff640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:50:10.112 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.112+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed5c004eb0 con 0x7fed7014be90 2024-03-25T19:50:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.118+0000 7fed57fff640 10 client.?.objecter ms_handle_connect 0x7fed700b4d60 2024-03-25T19:50:10.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.118+0000 7fed57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.125+0000 7fed57fff640 10 client.?.objecter ms_handle_connect 0x7fed700b1730 2024-03-25T19:50:10.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.125+0000 7fed57fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.126+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229369331 0 0) 0x7fed70141f40 con 0x7fed700b4d60 2024-03-25T19:50:10.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.126+0000 7fed57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.126+0000 7fed57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.126+0000 7fed57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.126+0000 7fed57fff640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:50:10.126 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.126+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed70142360 con 0x7fed700b4d60 2024-03-25T19:50:10.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.128+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359522904 0 0) 0x7fed5c004eb0 con 0x7fed7014be90 2024-03-25T19:50:10.128 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.128+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed70141f40 con 0x7fed7014be90 2024-03-25T19:50:10.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.132+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437551194 0 0) 0x7fed700b1030 con 0x7fed700b1730 2024-03-25T19:50:10.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.132+0000 7fed57fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.132+0000 7fed57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.132+0000 7fed57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.132 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.132+0000 7fed57fff640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:50:10.132 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.132+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed5c004eb0 con 0x7fed700b1730 2024-03-25T19:50:10.136 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.137+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed68001cd0 con 0x7fed7014be90 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.137+0000 7fed57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.137+0000 7fed57fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.137+0000 7fed57fff640 10 monclient(hunting): dump: 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:10.137 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.137+0000 7fed57fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:10.142 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.142+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 881846113 0 0) 0x7fed70142360 con 0x7fed700b4d60 2024-03-25T19:50:10.142 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.142+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed700b1030 con 0x7fed700b4d60 2024-03-25T19:50:10.142 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.142+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed640018b0 con 0x7fed700b4d60 2024-03-25T19:50:10.142 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.142+0000 7fed57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.142+0000 7fed57fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.142+0000 7fed57fff640 10 monclient(hunting): dump: 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:10.143 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2292815859 0 0) 0x7fed70141f40 con 0x7fed7014be90 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 >> v1:172.21.15.50:6790/0 conn(0x7fed700b1730 legacy=0x7fed701405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 >> v1:172.21.15.50:6789/0 conn(0x7fed700b4d60 legacy=0x7fed701d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 1 monclient: found mon.b 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70140d00 con 0x7fed7014be90 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:50:10.148 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 10 monclient: _renew_subs 2024-03-25T19:50:10.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:10.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed70138150 con 0x7fed7014be90 2024-03-25T19:50:10.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:10.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 10 monclient: _renew_subs 2024-03-25T19:50:10.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:10.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701d8520 con 0x7fed7014be90 2024-03-25T19:50:10.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.148+0000 7fed75e65640 1 librados: init done 2024-03-25T19:50:10.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.149+0000 7fed75e65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.149+0000 7fed75e65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.149+0000 7fed75e65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:10.150 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.149+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed70140f50 con 0x7fed7014be90 2024-03-25T19:50:10.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.428+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed68003970 con 0x7fed7014be90 2024-03-25T19:50:10.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.428+0000 7fed57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.428+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed68003dc0 con 0x7fed7014be90 2024-03-25T19:50:10.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.428+0000 7fed57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.428 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.428+0000 7fed57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.428+0000 7fed57fff640 10 monclient: dump: 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:10.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.428+0000 7fed6d7fa640 4 set_mon_vals no callback set 2024-03-25T19:50:10.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.429+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fed68003fe0 con 0x7fed7014be90 2024-03-25T19:50:10.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.430+0000 7fed6f7fe640 10 monclient: get_auth_request con 0x7fed5c06df90 auth_method 0 2024-03-25T19:50:10.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.430+0000 7fed57fff640 10 client.4636.objecter ms_handle_connect 0x7fed5c06df90 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7fed680549b0 con 0x7fed7014be90 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 10 client.4636.objecter ms_dispatch 0x7fed70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed680504c0 con 0x7fed7014be90 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.436 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.436+0000 7fed57fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:10.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.683+0000 7fed75e65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.683+0000 7fed75e65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.683+0000 7fed75e65640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:10.683 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.683+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed701d7950 con 0x7fed7014be90 2024-03-25T19:50:10.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.711+0000 7fed57fff640 1 -- 172.21.15.50:0/3878843441 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7fed701d7950 con 0x7fed7014be90 2024-03-25T19:50:10.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.711+0000 7fed57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.711+0000 7fed57fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:10.711 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:10.711 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:10.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 10 librados: watch_flush enter 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 10 librados: watch_flush exit 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 10 client.4636.objecter successfully canceled tick 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 >> v1:172.21.15.76:6800/2572556811 conn(0x7fed5c06df90 legacy=0x7fed5c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 10 monclient: shutdown 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 >> v1:172.21.15.76:6789/0 conn(0x7fed7014be90 legacy=0x7fed701e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 shutdown_connections 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.713+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 >> 172.21.15.50:0/3878843441 conn(0x7fed700617a0 msgr2=0x7fed700b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.714+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 shutdown_connections 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.714+0000 7fed75e65640 1 -- 172.21.15.50:0/3878843441 wait complete. 2024-03-25T19:50:10.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:10.714+0000 7fed75e65640 1 librados: shutdown 2024-03-25T19:50:10.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:17.533 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:17.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.654+0000 7f5fc8ee0640 10 monclient: get_monmap_and_config 2024-03-25T19:50:17.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.654+0000 7f5fc8ee0640 10 monclient: build_initial_monmap 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.654+0000 7f5fc8ee0640 10 monclient: monmap: 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:17.654874+0000 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:17.654874+0000 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:17.654 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:17.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.654+0000 7f5fc8ee0640 1 Processor -- start 2024-03-25T19:50:17.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.654+0000 7f5fc8ee0640 1 -- start start 2024-03-25T19:50:17.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.654+0000 7f5fc8ee0640 10 monclient: init 2024-03-25T19:50:17.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:17.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:17.655 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f5fc40a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:17.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:17.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f5fc41430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:17.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc4142520 con 0x7f5fc413f710 2024-03-25T19:50:17.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc40b28a0 con 0x7f5fc41430c0 2024-03-25T19:50:17.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc40b2a30 con 0x7f5fc40a98f0 2024-03-25T19:50:17.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:17.656 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.655+0000 7f5fc8ee0640 10 monclient(hunting): authenticate will time out at 5392.919520s 2024-03-25T19:50:17.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.659+0000 7f5fc1d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5fc413f710 0x7f5fc41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54344/0 (socket says 172.21.15.50:54344) 2024-03-25T19:50:17.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.659+0000 7f5fc1d9b640 1 -- 172.21.15.50:0/1548399558 learned_addr learned my addr 172.21.15.50:0/1548399558 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:17.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898089731 0 0) 0x7f5fc4142520 con 0x7f5fc413f710 2024-03-25T19:50:17.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:50:17.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb0003520 con 0x7f5fc413f710 2024-03-25T19:50:17.681 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195972886 0 0) 0x7f5fc40b28a0 con 0x7f5fc41430c0 2024-03-25T19:50:17.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:50:17.682 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.681+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc4142520 con 0x7f5fc41430c0 2024-03-25T19:50:17.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.685+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859204299 0 0) 0x7f5fc40b2a30 con 0x7f5fc40a98f0 2024-03-25T19:50:17.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.685+0000 7f5fc159a640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.685+0000 7f5fc159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.685+0000 7f5fc159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.685+0000 7f5fc159a640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:50:17.685 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.685+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc40b28a0 con 0x7f5fc40a98f0 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.697+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 398407508 0 0) 0x7f5fc40b28a0 con 0x7f5fc40a98f0 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 >> v1:172.21.15.76:6789/0 conn(0x7f5fc41430c0 legacy=0x7f5fc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 >> v1:172.21.15.50:6789/0 conn(0x7f5fc413f710 legacy=0x7f5fc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 10 monclient: _finish_hunting 0 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 1 monclient: found mon.noname-b 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc40b44c0 con 0x7f5fc40a98f0 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 10 monclient: _finish_auth 0 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 10 monclient: _check_auth_tickets 2024-03-25T19:50:17.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5fb40015b0 con 0x7f5fc40a98f0 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc8ee0640 5 monclient: authenticate success, global_id 4634 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc8ee0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc159a640 10 monclient: dump: 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:17.699 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:17.700 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:17.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.698+0000 7f5fc8ee0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:17.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fb4001710 con 0x7f5fc40a98f0 2024-03-25T19:50:17.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:17.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc159a640 1 -- 172.21.15.50:0/1548399558 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5fb4001b60 con 0x7f5fc40a98f0 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc159a640 10 monclient: dump: 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:17.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc8ee0640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:17.709 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc8ee0640 4 set_mon_vals no callback set 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc8ee0640 10 monclient: shutdown 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc8ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.708+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1548399558 >> v1:172.21.15.50:6790/0 conn(0x7f5fc40a98f0 legacy=0x7f5fc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.709+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1548399558 shutdown_connections 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.709+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1548399558 >> 172.21.15.50:0/1548399558 conn(0x7f5fc40b7ea0 msgr2=0x7f5fc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.709+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1548399558 shutdown_connections 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.709+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1548399558 wait complete. 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.709+0000 7f5fc8ee0640 10 monclient: build_initial_monmap 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.709+0000 7f5fc8ee0640 10 monclient: monmap: 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:17.710 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.710+0000 7f5fc8ee0640 1 librados: starting msgr at 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.710+0000 7f5fc8ee0640 1 librados: starting objecter 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.710+0000 7f5fc8ee0640 1 Processor -- start 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.710+0000 7f5fc8ee0640 1 -- start start 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.710+0000 7f5fc8ee0640 1 librados: setting wanted keys 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.710+0000 7f5fc8ee0640 1 librados: calling monclient init 2024-03-25T19:50:17.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.710+0000 7f5fc8ee0640 10 monclient: init 2024-03-25T19:50:17.712 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:17.712 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f5fc40a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f5fc413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 10 monclient(hunting): picked mon.noname-a con 0x7f5fc41430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc41394b0 con 0x7f5fc41430c0 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc4139720 con 0x7f5fc413f710 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc414e760 con 0x7f5fc40a98f0 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:17.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.711+0000 7f5fc8ee0640 10 monclient(hunting): authenticate will time out at 5392.975160s 2024-03-25T19:50:17.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.723+0000 7f5fc2d9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f5fc41430c0 0x7f5fc41e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:54412/0 (socket says 172.21.15.50:54412) 2024-03-25T19:50:17.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.723+0000 7f5fc2d9d640 1 -- 172.21.15.50:0/1331439403 learned_addr learned my addr 172.21.15.50:0/1331439403 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:17.730 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.731+0000 7f5fbaffd640 10 client.?.objecter ms_handle_connect 0x7f5fc40a98f0 2024-03-25T19:50:17.730 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.731+0000 7f5fbaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.732+0000 7f5fbaffd640 10 client.?.objecter ms_handle_connect 0x7f5fc41430c0 2024-03-25T19:50:17.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.732+0000 7f5fbaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.741+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257931255 0 0) 0x7f5fc41394b0 con 0x7f5fc41430c0 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.741+0000 7f5fbaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.741+0000 7f5fbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.741+0000 7f5fbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.741+0000 7f5fbaffd640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.741+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa4004eb0 con 0x7f5fc41430c0 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.742+0000 7f5fbaffd640 10 client.?.objecter ms_handle_connect 0x7f5fc413f710 2024-03-25T19:50:17.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.742+0000 7f5fbaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.747+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44732421 0 0) 0x7f5fc414e760 con 0x7f5fc40a98f0 2024-03-25T19:50:17.746 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.747+0000 7f5fbaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.747+0000 7f5fbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.747+0000 7f5fbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.747+0000 7f5fbaffd640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:50:17.747 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.747+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fc41394b0 con 0x7f5fc40a98f0 2024-03-25T19:50:17.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.758+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1740126207 0 0) 0x7f5fc41394b0 con 0x7f5fc40a98f0 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.759+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc414e760 con 0x7f5fc40a98f0 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.759+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5fb4001a20 con 0x7f5fc40a98f0 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.759+0000 7f5fbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.759+0000 7f5fbaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.759+0000 7f5fbaffd640 10 monclient(hunting): dump: 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:17.759 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:17.760 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:17.760 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.759+0000 7f5fbaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:17.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.761+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2251345373 0 0) 0x7f5fa4004eb0 con 0x7f5fc41430c0 2024-03-25T19:50:17.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.761+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fc41394b0 con 0x7f5fc41430c0 2024-03-25T19:50:17.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.761+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5fbc001c90 con 0x7f5fc41430c0 2024-03-25T19:50:17.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.761+0000 7f5fbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.761 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.761+0000 7f5fbaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.761+0000 7f5fbaffd640 10 monclient(hunting): dump: 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:17.762 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:17.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.770+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470641301 0 0) 0x7f5fc4139720 con 0x7f5fc413f710 2024-03-25T19:50:17.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.770+0000 7f5fbaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.770+0000 7f5fbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.770+0000 7f5fbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.770+0000 7f5fbaffd640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:50:17.770 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.770+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fa4004eb0 con 0x7f5fc413f710 2024-03-25T19:50:17.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1953864863 0 0) 0x7f5fc414e760 con 0x7f5fc40a98f0 2024-03-25T19:50:17.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 >> v1:172.21.15.76:6789/0 conn(0x7f5fc413f710 legacy=0x7f5fc41db090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 >> v1:172.21.15.50:6789/0 conn(0x7f5fc41430c0 legacy=0x7f5fc41e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:17.776 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 1 monclient: found mon.c 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc414e8f0 con 0x7f5fc40a98f0 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 5 monclient: authenticate success, global_id 4637 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 10 monclient: _renew_subs 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc4137610 con 0x7f5fc40a98f0 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 10 monclient: _renew_subs 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc4138390 con 0x7f5fc40a98f0 2024-03-25T19:50:17.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.776+0000 7f5fc8ee0640 1 librados: init done 2024-03-25T19:50:17.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.777+0000 7f5fc8ee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:17.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.777+0000 7f5fc8ee0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:17.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.777+0000 7f5fc8ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:17.778 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:17.777+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc40627b0 con 0x7f5fc40a98f0 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.091+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fb4001be0 con 0x7f5fc40a98f0 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.091+0000 7f5fbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.091+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f5fb4003a40 con 0x7f5fc40a98f0 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.091+0000 7f5fbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.091+0000 7f5fbaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.091+0000 7f5fbaffd640 10 monclient: dump: 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:18.091 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.091+0000 7f5fc0d99640 4 set_mon_vals no callback set 2024-03-25T19:50:18.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.092+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f5fb4019a10 con 0x7f5fc40a98f0 2024-03-25T19:50:18.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.092+0000 7f5fc1d9b640 10 monclient: get_auth_request con 0x7f5fa406df50 auth_method 0 2024-03-25T19:50:18.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.093+0000 7f5fbaffd640 10 client.4637.objecter ms_handle_connect 0x7f5fa406df50 2024-03-25T19:50:18.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f5fb4053610 con 0x7f5fc40a98f0 2024-03-25T19:50:18.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 10 client.4637.objecter ms_dispatch 0x7f5fc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:18.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:18.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:18.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-25T19:50:18.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fb404ef50 con 0x7f5fc40a98f0 2024-03-25T19:50:18.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:18.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.098+0000 7f5fbaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:18.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.361+0000 7f5fc8ee0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.361+0000 7f5fc8ee0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.361+0000 7f5fc8ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:18.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.361+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5fc4141bb0 con 0x7f5fc40a98f0 2024-03-25T19:50:18.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.388+0000 7f5fbaffd640 1 -- 172.21.15.50:0/1331439403 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f5fb4051050 con 0x7f5fc40a98f0 2024-03-25T19:50:18.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.388+0000 7f5fbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.388 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.388+0000 7f5fbaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:18.388 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:18.389 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 10 librados: watch_flush enter 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 10 librados: watch_flush exit 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 10 client.4637.objecter successfully canceled tick 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 >> v1:172.21.15.76:6800/2572556811 conn(0x7f5fa406df50 legacy=0x7f5fa4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 10 monclient: shutdown 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:18.391 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 >> v1:172.21.15.50:6790/0 conn(0x7f5fc40a98f0 legacy=0x7f5fc41da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 shutdown_connections 2024-03-25T19:50:18.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 >> 172.21.15.50:0/1331439403 conn(0x7f5fc40b7ea0 msgr2=0x7f5fc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:18.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 shutdown_connections 2024-03-25T19:50:18.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 1 -- 172.21.15.50:0/1331439403 wait complete. 2024-03-25T19:50:18.392 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:18.391+0000 7f5fc8ee0640 1 librados: shutdown 2024-03-25T19:50:18.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:21.101 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:41.607 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.107 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:41.613 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.149 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:09:41.655 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.163 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:41.669 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.173 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:41.678 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.194 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:41.665 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.203 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:09:41.709 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.208 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:41.678 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:21.255 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:09:41.726 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:25.209 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:25.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.325+0000 7f2893a06640 10 monclient: get_monmap_and_config 2024-03-25T19:50:25.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.325+0000 7f2893a06640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.325+0000 7f2893a06640 10 monclient: monmap: 2024-03-25T19:50:25.325 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:25.325 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:25.326603+0000 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:25.326603+0000 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:25.326 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:25.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.326+0000 7f2893a06640 1 Processor -- start 2024-03-25T19:50:25.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.326+0000 7f2893a06640 1 -- start start 2024-03-25T19:50:25.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.326+0000 7f2893a06640 10 monclient: init 2024-03-25T19:50:25.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.327 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 10 monclient(hunting): picked mon.noname-b con 0x7f288c0b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 10 monclient(hunting): picked mon.noname-a con 0x7f288c0b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 10 monclient(hunting): picked mon.noname-c con 0x7f288c14be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c0af940 con 0x7f288c0b1730 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c0b0a70 con 0x7f288c14be90 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c0b0d50 con 0x7f288c0b4d60 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.328 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.327+0000 7f2893a06640 10 monclient(hunting): authenticate will time out at 5400.591158s 2024-03-25T19:50:25.334 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.335+0000 7f2891fa3640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f288c14be90 0x7f288c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:48464/0 (socket says 172.21.15.50:48464) 2024-03-25T19:50:25.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.335+0000 7f2891fa3640 1 -- 172.21.15.50:0/1630344129 learned_addr learned my addr 172.21.15.50:0/1630344129 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:25.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.357+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282761873 0 0) 0x7f288c0b0a70 con 0x7f288c14be90 2024-03-25T19:50:25.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.357+0000 7f2883fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.357+0000 7f2883fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.357+0000 7f2883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.357+0000 7f2883fff640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:50:25.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.357+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2878003520 con 0x7f288c14be90 2024-03-25T19:50:25.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.359+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044742822 0 0) 0x7f288c0b0d50 con 0x7f288c0b4d60 2024-03-25T19:50:25.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.359+0000 7f2883fff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.359+0000 7f2883fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.359+0000 7f2883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.359+0000 7f2883fff640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:50:25.359 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.359+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f288c0b0a70 con 0x7f288c0b4d60 2024-03-25T19:50:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 546541390 0 0) 0x7f288c0b0a70 con 0x7f288c0b4d60 2024-03-25T19:50:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 >> v1:172.21.15.76:6789/0 conn(0x7f288c14be90 legacy=0x7f288c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 >> v1:172.21.15.50:6789/0 conn(0x7f288c0b1730 legacy=0x7f288c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 1 monclient: found mon.noname-b 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c0b1030 con 0x7f288c0b4d60 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2888001670 con 0x7f288c0b4d60 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2893a06640 5 monclient: authenticate success, global_id 4640 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2893a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.372 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2883fff640 10 monclient: dump: 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:25.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.371+0000 7f2893a06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:25.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28880017d0 con 0x7f288c0b4d60 2024-03-25T19:50:25.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2883fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:25.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2883fff640 1 -- 172.21.15.50:0/1630344129 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2888001ba0 con 0x7f288c0b4d60 2024-03-25T19:50:25.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2893a06640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:25.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2893a06640 4 set_mon_vals no callback set 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2883fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2883fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.379+0000 7f2883fff640 10 monclient: dump: 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:25.380 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:25.381 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:25.381 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:25.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.380+0000 7f2893a06640 10 monclient: shutdown 2024-03-25T19:50:25.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.380+0000 7f2893a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:25.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.380+0000 7f2893a06640 1 -- 172.21.15.50:0/1630344129 >> v1:172.21.15.50:6790/0 conn(0x7f288c0b4d60 legacy=0x7f288c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.380+0000 7f2893a06640 1 -- 172.21.15.50:0/1630344129 shutdown_connections 2024-03-25T19:50:25.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.380+0000 7f2893a06640 1 -- 172.21.15.50:0/1630344129 >> 172.21.15.50:0/1630344129 conn(0x7f288c0617a0 msgr2=0x7f288c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:25.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.380+0000 7f2893a06640 1 -- 172.21.15.50:0/1630344129 shutdown_connections 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.380+0000 7f2893a06640 1 -- 172.21.15.50:0/1630344129 wait complete. 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.381+0000 7f2893a06640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.381+0000 7f2893a06640 10 monclient: monmap: 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:25.384 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.381+0000 7f2893a06640 1 librados: starting msgr at 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.381+0000 7f2893a06640 1 librados: starting objecter 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.381+0000 7f2893a06640 1 Processor -- start 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 1 -- start start 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 1 librados: setting wanted keys 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 1 librados: calling monclient init 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient: init 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient(hunting): picked mon.noname-b con 0x7f288c0b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient(hunting): picked mon.noname-a con 0x7f288c0b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient(hunting): picked mon.noname-c con 0x7f288c14be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c141c80 con 0x7f288c0b4d60 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c137d20 con 0x7f288c0b1730 2024-03-25T19:50:25.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288c140a20 con 0x7f288c14be90 2024-03-25T19:50:25.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.382+0000 7f2893a06640 10 monclient(hunting): authenticate will time out at 5400.646444s 2024-03-25T19:50:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.395+0000 7f2890fa1640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f288c0b4d60 0x7f288c1dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58794/0 (socket says 172.21.15.50:58794) 2024-03-25T19:50:25.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.395+0000 7f2890fa1640 1 -- 172.21.15.50:0/385984254 learned_addr learned my addr 172.21.15.50:0/385984254 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:25.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.400+0000 7f2881ffb640 10 client.?.objecter ms_handle_connect 0x7f288c14be90 2024-03-25T19:50:25.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.400+0000 7f2881ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.405+0000 7f2881ffb640 10 client.?.objecter ms_handle_connect 0x7f288c0b1730 2024-03-25T19:50:25.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.405+0000 7f2881ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.405+0000 7f2881ffb640 10 client.?.objecter ms_handle_connect 0x7f288c0b4d60 2024-03-25T19:50:25.405 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.405+0000 7f2881ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.413+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716741982 0 0) 0x7f288c137d20 con 0x7f288c0b1730 2024-03-25T19:50:25.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.413+0000 7f2881ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.413+0000 7f2881ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.413+0000 7f2881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.413+0000 7f2881ffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:50:25.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.413+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2870004eb0 con 0x7f288c0b1730 2024-03-25T19:50:25.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.419+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871323137 0 0) 0x7f288c140a20 con 0x7f288c14be90 2024-03-25T19:50:25.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.419+0000 7f2881ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.419+0000 7f2881ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.419+0000 7f2881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.419+0000 7f2881ffb640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:50:25.419 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.419+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28700051d0 con 0x7f288c14be90 2024-03-25T19:50:25.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.420+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327424881 0 0) 0x7f288c141c80 con 0x7f288c0b4d60 2024-03-25T19:50:25.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.420+0000 7f2881ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.420+0000 7f2881ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.420+0000 7f2881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.420+0000 7f2881ffb640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:50:25.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.420+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f288c140a20 con 0x7f288c0b4d60 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.422+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843558882 0 0) 0x7f2870004eb0 con 0x7f288c0b1730 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.422+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f288c141c80 con 0x7f288c0b1730 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.422+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2888001ae0 con 0x7f288c0b1730 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.422+0000 7f2881ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.422+0000 7f2881ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.422+0000 7f2881ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.422 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:25.423 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:25.423 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:25.423 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:25.423 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:25.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.422+0000 7f2881ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:25.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.429+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3757152720 0 0) 0x7f288c140a20 con 0x7f288c0b4d60 2024-03-25T19:50:25.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.430+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2870004eb0 con 0x7f288c0b4d60 2024-03-25T19:50:25.429 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.430+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f287c001930 con 0x7f288c0b4d60 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.430+0000 7f2881ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.430+0000 7f2881ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.430+0000 7f2881ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:25.430 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863850283 0 0) 0x7f288c141c80 con 0x7f288c0b1730 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 >> v1:172.21.15.50:6790/0 conn(0x7f288c14be90 legacy=0x7f288c1e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 >> v1:172.21.15.50:6789/0 conn(0x7f288c0b4d60 legacy=0x7f288c1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 1 monclient: found mon.b 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c132620 con 0x7f288c0b1730 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2881ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 10 monclient: _renew_subs 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288c132810 con 0x7f288c0b1730 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 10 monclient: _renew_subs 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c1db4d0 con 0x7f288c0b1730 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.433+0000 7f2893a06640 1 librados: init done 2024-03-25T19:50:25.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.434+0000 7f2893a06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.434+0000 7f2893a06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.434+0000 7f2893a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:25.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.434+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288c0b02f0 con 0x7f288c0b1730 2024-03-25T19:50:25.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.725+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28880038a0 con 0x7f288c0b1730 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.725+0000 7f2881ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.725+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2888003bb0 con 0x7f288c0b1730 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.725+0000 7f2881ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.725+0000 7f2881ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.725+0000 7f2881ffb640 10 monclient: dump: 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.726 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:25.727 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:25.727 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:25.727 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:25.727 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:25.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.725+0000 7f28837fe640 4 set_mon_vals no callback set 2024-03-25T19:50:25.727 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.726+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f2888019b80 con 0x7f288c0b1730 2024-03-25T19:50:25.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.727+0000 7f2890fa1640 10 monclient: get_auth_request con 0x7f287006e0c0 auth_method 0 2024-03-25T19:50:25.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.727+0000 7f2881ffb640 10 client.4648.objecter ms_handle_connect 0x7f287006e0c0 2024-03-25T19:50:25.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.731+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f2888053640 con 0x7f288c0b1730 2024-03-25T19:50:25.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.731+0000 7f2881ffb640 10 client.4648.objecter ms_dispatch 0x7f288c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:25.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.731+0000 7f2881ffb640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:25.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.731+0000 7f2881ffb640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:25.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.731+0000 7f2881ffb640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:50:25.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.732+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f288804f150 con 0x7f288c0b1730 2024-03-25T19:50:25.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.732+0000 7f2881ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.732+0000 7f2881ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:25.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.978+0000 7f2893a06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:25.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.978+0000 7f2893a06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:25.978 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.978+0000 7f2893a06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:25.979 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:25.978+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f288c0627b0 con 0x7f288c0b1730 2024-03-25T19:50:26.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.048+0000 7f2881ffb640 1 -- 172.21.15.50:0/385984254 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f2888051080 con 0x7f288c0b1730 2024-03-25T19:50:26.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.048+0000 7f2881ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.048+0000 7f2881ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:26.048 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:26.048 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.050+0000 7f2893a06640 10 librados: watch_flush enter 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.050+0000 7f2893a06640 10 librados: watch_flush exit 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.050+0000 7f2893a06640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.050+0000 7f2893a06640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 >> v1:172.21.15.76:6800/2572556811 conn(0x7f287006e0c0 legacy=0x7f2870070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 10 monclient: shutdown 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 >> v1:172.21.15.76:6789/0 conn(0x7f288c0b1730 legacy=0x7f288c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 shutdown_connections 2024-03-25T19:50:26.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 >> 172.21.15.50:0/385984254 conn(0x7f288c0617a0 msgr2=0x7f288c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:26.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 shutdown_connections 2024-03-25T19:50:26.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 1 -- 172.21.15.50:0/385984254 wait complete. 2024-03-25T19:50:26.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:26.051+0000 7f2893a06640 1 librados: shutdown 2024-03-25T19:50:26.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:32.870 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.965+0000 7f761bcc1640 10 monclient: get_monmap_and_config 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.965+0000 7f761bcc1640 10 monclient: build_initial_monmap 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.965+0000 7f761bcc1640 10 monclient: monmap: 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:32.967446+0000 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:32.967446+0000 2024-03-25T19:50:32.966 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:32.967 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:32.967 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:32.967 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:32.967 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:32.967 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 1 Processor -- start 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 1 -- start start 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 10 monclient: init 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f76140b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f76140b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.966+0000 7f761bcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f761414be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.967+0000 7f761bcc1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76140af940 con 0x7f761414be90 2024-03-25T19:50:32.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.967+0000 7f761bcc1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76140b0a70 con 0x7f76140b1730 2024-03-25T19:50:32.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.967+0000 7f761bcc1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76140b0d50 con 0x7f76140b4d60 2024-03-25T19:50:32.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.967+0000 7f761bcc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:32.969 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.967+0000 7f761bcc1640 10 monclient(hunting): authenticate will time out at 5408.231737s 2024-03-25T19:50:32.981 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.981+0000 7f761a25e640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f761414be90 0x7f761414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:34516/0 (socket says 172.21.15.50:34516) 2024-03-25T19:50:32.982 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:32.981+0000 7f761a25e640 1 -- 172.21.15.50:0/1665449801 learned_addr learned my addr 172.21.15.50:0/1665449801 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:33.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.007+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436098174 0 0) 0x7f76140af940 con 0x7f761414be90 2024-03-25T19:50:33.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.007+0000 7f7618a5b640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.007+0000 7f7618a5b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.007+0000 7f7618a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.007+0000 7f7618a5b640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:50:33.008 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.007+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7600003520 con 0x7f761414be90 2024-03-25T19:50:33.015 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588904956 0 0) 0x7f7600003520 con 0x7f761414be90 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 >> v1:172.21.15.50:6790/0 conn(0x7f76140b4d60 legacy=0x7f761413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 >> v1:172.21.15.76:6789/0 conn(0x7f76140b1730 legacy=0x7f76140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 1 monclient: found mon.noname-a 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76140b1030 con 0x7f761414be90 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f760c001a10 con 0x7f761414be90 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f761bcc1640 5 monclient: authenticate success, global_id 4674 2024-03-25T19:50:33.016 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f761bcc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f7618a5b640 10 monclient: dump: 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:33.017 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.014+0000 7f761bcc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:33.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.043+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f760c001be0 con 0x7f761414be90 2024-03-25T19:50:33.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f7618a5b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f7618a5b640 1 -- 172.21.15.50:0/1665449801 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f760c002030 con 0x7f761414be90 2024-03-25T19:50:33.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f7618a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f7618a5b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f7618a5b640 10 monclient: dump: 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:33.045 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f761bcc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:33.046 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f761bcc1640 4 set_mon_vals no callback set 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f761bcc1640 10 monclient: shutdown 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f761bcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f761bcc1640 1 -- 172.21.15.50:0/1665449801 >> v1:172.21.15.50:6789/0 conn(0x7f761414be90 legacy=0x7f761414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f761bcc1640 1 -- 172.21.15.50:0/1665449801 shutdown_connections 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.044+0000 7f761bcc1640 1 -- 172.21.15.50:0/1665449801 >> 172.21.15.50:0/1665449801 conn(0x7f76140617a0 msgr2=0x7f7614061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.045+0000 7f761bcc1640 1 -- 172.21.15.50:0/1665449801 shutdown_connections 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.045+0000 7f761bcc1640 1 -- 172.21.15.50:0/1665449801 wait complete. 2024-03-25T19:50:33.047 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 10 monclient: build_initial_monmap 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 10 monclient: monmap: 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 1 librados: starting msgr at 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 1 librados: starting objecter 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 1 Processor -- start 2024-03-25T19:50:33.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 1 -- start start 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 1 librados: setting wanted keys 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 1 librados: calling monclient init 2024-03-25T19:50:33.049 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.046+0000 7f761bcc1640 10 monclient: init 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 10 monclient(hunting): picked mon.noname-a con 0x7f76140b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 10 monclient(hunting): picked mon.noname-b con 0x7f76140b4d60 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 10 monclient(hunting): picked mon.noname-c con 0x7f761414be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76141399e0 con 0x7f76140b1730 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7614137b70 con 0x7f76140b4d60 2024-03-25T19:50:33.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7614138370 con 0x7f761414be90 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:33.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.047+0000 7f761bcc1640 10 monclient(hunting): authenticate will time out at 5408.312345s 2024-03-25T19:50:33.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.055+0000 7f761925c640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f76140b4d60 0x7f761413ae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:49114/0 (socket says 172.21.15.50:49114) 2024-03-25T19:50:33.056 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.056+0000 7f761925c640 1 -- 172.21.15.50:0/4092640972 learned_addr learned my addr 172.21.15.50:0/4092640972 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:33.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.059+0000 7f760a7fc640 10 client.?.objecter ms_handle_connect 0x7f76140b4d60 2024-03-25T19:50:33.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.059+0000 7f760a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.060+0000 7f760a7fc640 10 client.?.objecter ms_handle_connect 0x7f761414be90 2024-03-25T19:50:33.060 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.060+0000 7f760a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.062+0000 7f760a7fc640 10 client.?.objecter ms_handle_connect 0x7f76140b1730 2024-03-25T19:50:33.062 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.062+0000 7f760a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.065+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601743141 0 0) 0x7f76141399e0 con 0x7f76140b1730 2024-03-25T19:50:33.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.065+0000 7f760a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.065 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.065+0000 7f760a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.065+0000 7f760a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.065+0000 7f760a7fc640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:50:33.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.065+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f8004eb0 con 0x7f76140b1730 2024-03-25T19:50:33.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.066+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129016016 0 0) 0x7f7614137b70 con 0x7f76140b4d60 2024-03-25T19:50:33.066 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.066+0000 7f760a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.066+0000 7f760a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.066+0000 7f760a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.066+0000 7f760a7fc640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:50:33.067 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.066+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76141399e0 con 0x7f76140b4d60 2024-03-25T19:50:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.071+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010792193 0 0) 0x7f7614138370 con 0x7f761414be90 2024-03-25T19:50:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.072+0000 7f760a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.072+0000 7f760a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.072+0000 7f760a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.072+0000 7f760a7fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:50:33.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.072+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7614137b70 con 0x7f761414be90 2024-03-25T19:50:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.077+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2829463713 0 0) 0x7f75f8004eb0 con 0x7f76140b1730 2024-03-25T19:50:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.077+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7614138370 con 0x7f76140b1730 2024-03-25T19:50:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.077+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7610001870 con 0x7f76140b1730 2024-03-25T19:50:33.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.077+0000 7f760a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.077+0000 7f760a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.077+0000 7f760a7fc640 10 monclient(hunting): dump: 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:33.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.077+0000 7f760a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:33.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1106574151 0 0) 0x7f7614138370 con 0x7f76140b1730 2024-03-25T19:50:33.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 >> v1:172.21.15.50:6790/0 conn(0x7f761414be90 legacy=0x7f761413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 >> v1:172.21.15.76:6789/0 conn(0x7f76140b4d60 legacy=0x7f761413ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 1 monclient: found mon.a 2024-03-25T19:50:33.089 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614132520 con 0x7f76140b1730 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f760a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 5 monclient: authenticate success, global_id 4677 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 10 monclient: _renew_subs 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76141327d0 con 0x7f76140b1730 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 10 monclient: _renew_subs 2024-03-25T19:50:33.090 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:50:33.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7614145260 con 0x7f76140b1730 2024-03-25T19:50:33.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.088+0000 7f761bcc1640 1 librados: init done 2024-03-25T19:50:33.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.089+0000 7f761bcc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.089+0000 7f761bcc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.089+0000 7f761bcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:50:33.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.089+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76140627b0 con 0x7f76140b1730 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76100035f0 con 0x7f76140b1730 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7610001d80 con 0x7f76140b1730 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760a7fc640 10 monclient: dump: 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:33.406 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760bfff640 4 set_mon_vals no callback set 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.405+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f76100197b0 con 0x7f76140b1730 2024-03-25T19:50:33.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.406+0000 7f761925c640 10 monclient: get_auth_request con 0x7f75f806ddf0 auth_method 0 2024-03-25T19:50:33.408 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.406+0000 7f760a7fc640 10 client.4677.objecter ms_handle_connect 0x7f75f806ddf0 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f7610053550 con 0x7f76140b1730 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 10 client.4677.objecter ms_dispatch 0x7f7614000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 3 client.4677.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 3 client.4677.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f761004ee90 con 0x7f76140b1730 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.412+0000 7f760a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:33.666 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.666+0000 7f761bcc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.666+0000 7f761bcc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.666+0000 7f761bcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:50:33.667 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.666+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7614069f40 con 0x7f76140b1730 2024-03-25T19:50:33.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.697+0000 7f760a7fc640 1 -- 172.21.15.50:0/4092640972 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f7610050f90 con 0x7f76140b1730 2024-03-25T19:50:33.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.697+0000 7f760a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.697+0000 7f760a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:33.698 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:33.698 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:33.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 10 librados: watch_flush enter 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 10 librados: watch_flush exit 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 10 client.4677.objecter successfully canceled tick 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 >> v1:172.21.15.76:6800/2572556811 conn(0x7f75f806ddf0 legacy=0x7f75f80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 10 monclient: shutdown 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 >> v1:172.21.15.50:6789/0 conn(0x7f76140b1730 legacy=0x7f7614131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 shutdown_connections 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 >> 172.21.15.50:0/4092640972 conn(0x7f76140617a0 msgr2=0x7f761413c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 shutdown_connections 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 1 -- 172.21.15.50:0/4092640972 wait complete. 2024-03-25T19:50:33.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:33.699+0000 7f761bcc1640 1 librados: shutdown 2024-03-25T19:50:33.712 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:40.518 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:40.636 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.637+0000 7f955b274640 10 monclient: get_monmap_and_config 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.637+0000 7f955b274640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.637+0000 7f955b274640 10 monclient: monmap: 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:40.638082+0000 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:40.638082+0000 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:40.637 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.638+0000 7f955b274640 1 Processor -- start 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.638+0000 7f955b274640 1 -- start start 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.638+0000 7f955b274640 10 monclient: init 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 10 monclient(hunting): picked mon.noname-b con 0x7f9554070410 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 10 monclient(hunting): picked mon.noname-a con 0x7f95540707f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 10 monclient(hunting): picked mon.noname-c con 0x7f955413db40 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540adea0 con 0x7f95540707f0 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95541445a0 con 0x7f955413db40 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540720a0 con 0x7f9554070410 2024-03-25T19:50:40.639 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.639+0000 7f955b274640 10 monclient(hunting): authenticate will time out at 5415.903101s 2024-03-25T19:50:40.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.645+0000 7f955880f640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f95540707f0 0x7f955413d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52360/0 (socket says 172.21.15.50:52360) 2024-03-25T19:50:40.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.645+0000 7f955880f640 1 -- 172.21.15.50:0/2825956388 learned_addr learned my addr 172.21.15.50:0/2825956388 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:40.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.654+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3769052469 0 0) 0x7f95541445a0 con 0x7f955413db40 2024-03-25T19:50:40.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.654+0000 7f954bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.654+0000 7f954bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.654+0000 7f954bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.654+0000 7f954bfff640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:50:40.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.654+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9540003520 con 0x7f955413db40 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201112609 0 0) 0x7f95540adea0 con 0x7f95540707f0 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95541445a0 con 0x7f95540707f0 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412159050 0 0) 0x7f95540720a0 con 0x7f9554070410 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:50:40.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.658+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95540adea0 con 0x7f9554070410 2024-03-25T19:50:40.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4048185088 0 0) 0x7f95540adea0 con 0x7f9554070410 2024-03-25T19:50:40.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 >> v1:172.21.15.76:6789/0 conn(0x7f955413db40 legacy=0x7f955414ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.668 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 >> v1:172.21.15.50:6789/0 conn(0x7f95540707f0 legacy=0x7f955413d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 1 monclient: found mon.noname-b 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9554072380 con 0x7f9554070410 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 10 monclient: _finish_auth 0 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f95500015b0 con 0x7f9554070410 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.668+0000 7f954bfff640 10 monclient: dump: 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:40.669 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.669+0000 7f955b274640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:50:40.670 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.669+0000 7f955b274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9550001710 con 0x7f9554070410 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f954bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f954bfff640 1 -- 172.21.15.50:0/2825956388 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9550001b60 con 0x7f9554070410 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f954bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f954bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f954bfff640 10 monclient: dump: 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:40.691 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f955b274640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:40.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.691+0000 7f955b274640 4 set_mon_vals no callback set 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.692+0000 7f955b274640 10 monclient: shutdown 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.692+0000 7f955b274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.692+0000 7f955b274640 1 -- 172.21.15.50:0/2825956388 >> v1:172.21.15.50:6790/0 conn(0x7f9554070410 legacy=0x7f9554143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.692+0000 7f955b274640 1 -- 172.21.15.50:0/2825956388 shutdown_connections 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.692+0000 7f955b274640 1 -- 172.21.15.50:0/2825956388 >> 172.21.15.50:0/2825956388 conn(0x7f955406b630 msgr2=0x7f95540b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.692+0000 7f955b274640 1 -- 172.21.15.50:0/2825956388 shutdown_connections 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.692+0000 7f955b274640 1 -- 172.21.15.50:0/2825956388 wait complete. 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 10 monclient: monmap: 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:40.694 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:40.695 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 1 librados: starting msgr at 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 1 librados: starting objecter 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 1 Processor -- start 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 1 -- start start 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 1 librados: setting wanted keys 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 1 librados: calling monclient init 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.694+0000 7f955b274640 10 monclient: init 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 10 monclient(hunting): picked mon.noname-c con 0x7f9554070410 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 10 monclient(hunting): picked mon.noname-a con 0x7f95540707f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 10 monclient(hunting): picked mon.noname-b con 0x7f955413db40 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95541d5560 con 0x7f95540707f0 2024-03-25T19:50:40.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9554141270 con 0x7f955413db40 2024-03-25T19:50:40.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540ab290 con 0x7f9554070410 2024-03-25T19:50:40.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.695+0000 7f955b274640 10 monclient(hunting): authenticate will time out at 5415.959577s 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.704+0000 7f955880f640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f95540707f0 0x7f95541d59f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:52416/0 (socket says 172.21.15.50:52416) 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.704+0000 7f955880f640 1 -- 172.21.15.50:0/1615323864 learned_addr learned my addr 172.21.15.50:0/1615323864 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.705+0000 7f9549ffb640 10 client.?.objecter ms_handle_connect 0x7f95540707f0 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.705+0000 7f9549ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.705+0000 7f9549ffb640 10 client.?.objecter ms_handle_connect 0x7f955413db40 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.705+0000 7f9549ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.706+0000 7f9549ffb640 10 client.?.objecter ms_handle_connect 0x7f9554070410 2024-03-25T19:50:40.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.706+0000 7f9549ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.721+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699477926 0 0) 0x7f9554141270 con 0x7f955413db40 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.721+0000 7f9549ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.721+0000 7f9549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.721+0000 7f9549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.721+0000 7f9549ffb640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:50:40.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.722+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9538004eb0 con 0x7f955413db40 2024-03-25T19:50:40.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.722+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287452784 0 0) 0x7f95541d5560 con 0x7f95540707f0 2024-03-25T19:50:40.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.722+0000 7f9549ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.722 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.722+0000 7f9549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.722+0000 7f9549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.722+0000 7f9549ffb640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:50:40.723 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.722+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9554141270 con 0x7f95540707f0 2024-03-25T19:50:40.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.725+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814423191 0 0) 0x7f95540ab290 con 0x7f9554070410 2024-03-25T19:50:40.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.725+0000 7f9549ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.725+0000 7f9549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.725+0000 7f9549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.725+0000 7f9549ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:50:40.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.725+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95541d5560 con 0x7f9554070410 2024-03-25T19:50:40.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.738+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1097434267 0 0) 0x7f9554141270 con 0x7f95540707f0 2024-03-25T19:50:40.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.738+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95540ab290 con 0x7f95540707f0 2024-03-25T19:50:40.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.738+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9544001970 con 0x7f95540707f0 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.738+0000 7f9549ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.738+0000 7f9549ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.738+0000 7f9549ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.738+0000 7f9549ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.741+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3544625231 0 0) 0x7f9538004eb0 con 0x7f955413db40 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.741+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9554141270 con 0x7f955413db40 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.741+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f954c001b30 con 0x7f955413db40 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.741+0000 7f9549ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.741+0000 7f9549ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.741+0000 7f9549ffb640 10 monclient(hunting): dump: 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:40.741 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:40.742 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:40.742 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:40.742 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:40.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3069799346 0 0) 0x7f9554141270 con 0x7f955413db40 2024-03-25T19:50:40.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.50:6790/0 conn(0x7f9554070410 legacy=0x7f9554140b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.50:6789/0 conn(0x7f95540707f0 legacy=0x7f95541d59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:40.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 1 monclient: found mon.b 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95541d6100 con 0x7f955413db40 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f9549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f955b274640 5 monclient: authenticate success, global_id 4660 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f955b274640 10 monclient: _renew_subs 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f955b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.754+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95541d62f0 con 0x7f955413db40 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 10 monclient: _renew_subs 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:40.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f955413e200 con 0x7f955413db40 2024-03-25T19:50:40.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 1 librados: init done 2024-03-25T19:50:40.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:40.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:40.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.755+0000 7f955b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:40.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:40.756+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9554074d80 con 0x7f955413db40 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.030+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f954c002000 con 0x7f955413db40 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.030+0000 7f9549ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.030+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f954c003990 con 0x7f955413db40 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.030+0000 7f9549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.030+0000 7f9549ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:41.030 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.030+0000 7f9549ffb640 10 monclient: dump: 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:41.031 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.030+0000 7f954b7fe640 4 set_mon_vals no callback set 2024-03-25T19:50:41.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.031+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f954c019960 con 0x7f955413db40 2024-03-25T19:50:41.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.032+0000 7f9559010640 10 monclient: get_auth_request con 0x7f953806dfb0 auth_method 0 2024-03-25T19:50:41.032 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.032+0000 7f9549ffb640 10 client.4660.objecter ms_handle_connect 0x7f953806dfb0 2024-03-25T19:50:41.035 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f954c0534a0 con 0x7f955413db40 2024-03-25T19:50:41.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 10 client.4660.objecter ms_dispatch 0x7f9554000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:41.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:41.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:41.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T19:50:41.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f954c04ede0 con 0x7f955413db40 2024-03-25T19:50:41.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.036 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.036+0000 7f9549ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:41.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.258+0000 7f955b274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.258+0000 7f955b274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.258+0000 7f955b274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:41.258 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.258+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95541d5710 con 0x7f955413db40 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9559811640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.76:6789/0 conn(0x7f955413db40 legacy=0x7f95541dff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9559811640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.76:6789/0 conn(0x7f955413db40 legacy=0x7f95541dff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9559811640 1 --1- 172.21.15.50:0/1615323864 >> v1:172.21.15.76:6789/0 conn(0x7f955413db40 0x7f95541dff30 :-1 s=OPENED pgs=69970 cs=1 l=1).handle_message read tag failed 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9559811640 1 --1- 172.21.15.50:0/1615323864 >> v1:172.21.15.76:6789/0 conn(0x7f955413db40 0x7f95541dff30 :-1 s=OPENED pgs=69970 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.76:6789/0 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.76:6789/0 conn(0x7f955413db40 legacy=0x7f95541dff30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:50:41.270 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:41.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 10 monclient(hunting): picked mon.c con 0x7f9554070410 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:41.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 10 monclient(hunting): picked mon.a con 0x7f95540707f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:41.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 10 monclient(hunting): picked mon.b con 0x7f953807a470 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:41.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9554141270 con 0x7f95540707f0 2024-03-25T19:50:41.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9538004eb0 con 0x7f953807a470 2024-03-25T19:50:41.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f95380805b0 con 0x7f9554070410 2024-03-25T19:50:41.271 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.270+0000 7f9549ffb640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:41.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.285+0000 7f9549ffb640 10 client.4660.objecter ms_handle_connect 0x7f953807a470 2024-03-25T19:50:41.285 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.285+0000 7f9549ffb640 10 client.4660.objecter resend_mon_ops 2024-03-25T19:50:41.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.288+0000 7f9549ffb640 10 client.4660.objecter ms_handle_connect 0x7f95540707f0 2024-03-25T19:50:41.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.288+0000 7f9549ffb640 10 client.4660.objecter resend_mon_ops 2024-03-25T19:50:41.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.296+0000 7f9549ffb640 10 client.4660.objecter ms_handle_connect 0x7f9554070410 2024-03-25T19:50:41.295 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.296+0000 7f9549ffb640 10 client.4660.objecter resend_mon_ops 2024-03-25T19:50:41.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.296+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608729637 0 0) 0x7f9538004eb0 con 0x7f953807a470 2024-03-25T19:50:41.296 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.296+0000 7f9549ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:41.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.296+0000 7f9549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:41.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.296+0000 7f9549ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:50:41.297 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.296+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.76:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f953807e550 con 0x7f953807a470 2024-03-25T19:50:41.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.299+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411196543 0 0) 0x7f9554141270 con 0x7f95540707f0 2024-03-25T19:50:41.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.299+0000 7f9549ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:41.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.299+0000 7f9549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:41.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.299+0000 7f9549ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:50:41.299 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.299+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f9538004eb0 con 0x7f95540707f0 2024-03-25T19:50:41.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.305+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441835772 0 0) 0x7f95380805b0 con 0x7f9554070410 2024-03-25T19:50:41.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.305+0000 7f9549ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:41.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.305+0000 7f9549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:41.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.305+0000 7f9549ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:50:41.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.305+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f9554141270 con 0x7f9554070410 2024-03-25T19:50:41.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2296724447 0 0) 0x7f9538004eb0 con 0x7f95540707f0 2024-03-25T19:50:41.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.50:6790/0 conn(0x7f9554070410 legacy=0x7f9538074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.76:6789/0 conn(0x7f953807a470 legacy=0x7f953807c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:41.319 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 1 monclient: found mon.a 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f9538081810 con 0x7f95540707f0 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95380805b0 con 0x7f95540707f0 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f954c052da0 con 0x7f95540707f0 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:41.320 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.319+0000 7f9549ffb640 10 monclient: dump: 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:41.321 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:41.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.354+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f954c050ee0 con 0x7f95540707f0 2024-03-25T19:50:41.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.354+0000 7f9549ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:41.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.354+0000 7f954b7fe640 4 set_mon_vals no callback set 2024-03-25T19:50:41.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.355+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f954c04ea60 con 0x7f95540707f0 2024-03-25T19:50:41.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.380+0000 7f9549ffb640 1 -- 172.21.15.50:0/1615323864 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f954c001990 con 0x7f95540707f0 2024-03-25T19:50:41.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.380+0000 7f9549ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.379 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.380+0000 7f9549ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:41.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 10 librados: watch_flush enter 2024-03-25T19:50:41.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 10 librados: watch_flush exit 2024-03-25T19:50:41.382 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 10 client.4660.objecter successfully canceled tick 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.76:6800/2572556811 conn(0x7f953806dfb0 legacy=0x7f9538070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 10 monclient: shutdown 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 >> v1:172.21.15.50:6789/0 conn(0x7f95540707f0 legacy=0x7f9538078960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 shutdown_connections 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 >> 172.21.15.50:0/1615323864 conn(0x7f955406b630 msgr2=0x7f9554070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 shutdown_connections 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 1 -- 172.21.15.50:0/1615323864 wait complete. 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:41.382+0000 7f955b274640 1 librados: shutdown 2024-03-25T19:50:41.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:46.137 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:06.642 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.143 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:06.649 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.184 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:06.689 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.208 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:06.679 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.217 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:06.688 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.234 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:06.740 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.241 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:06.746 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.251 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:06.722 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:46.277 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:06.782 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:48.201 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.322+0000 7f4d68382640 10 monclient: get_monmap_and_config 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.322+0000 7f4d68382640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.322+0000 7f4d68382640 10 monclient: monmap: 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:48.323308+0000 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:48.323308+0000 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.322 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:48.323 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:48.323 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:48.323 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:48.323 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:48.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.323+0000 7f4d68382640 1 Processor -- start 2024-03-25T19:50:48.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.323+0000 7f4d68382640 1 -- start start 2024-03-25T19:50:48.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.323+0000 7f4d68382640 10 monclient: init 2024-03-25T19:50:48.324 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 10 monclient(hunting): picked mon.noname-c con 0x7f4d60149f50 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 10 monclient(hunting): picked mon.noname-a con 0x7f4d6006c780 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 10 monclient(hunting): picked mon.noname-b con 0x7f4d600b1b10 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6014a330 con 0x7f4d6006c780 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d60070660 con 0x7f4d60149f50 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6006d7c0 con 0x7f4d600b1b10 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.324+0000 7f4d68382640 10 monclient(hunting): authenticate will time out at 5423.588140s 2024-03-25T19:50:48.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.332+0000 7f4d6611e640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4d60149f50 0x7f4d600b0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:41544/0 (socket says 172.21.15.50:41544) 2024-03-25T19:50:48.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.332+0000 7f4d6611e640 1 -- 172.21.15.50:0/3511540694 learned_addr learned my addr 172.21.15.50:0/3511540694 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:48.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.347+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344725209 0 0) 0x7f4d60070660 con 0x7f4d60149f50 2024-03-25T19:50:48.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.347+0000 7f4d6511c640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.347+0000 7f4d6511c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.347+0000 7f4d6511c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.347+0000 7f4d6511c640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:50:48.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.347+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d4c003520 con 0x7f4d60149f50 2024-03-25T19:50:48.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.353+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688540643 0 0) 0x7f4d6014a330 con 0x7f4d6006c780 2024-03-25T19:50:48.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.353+0000 7f4d6511c640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.353 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.353+0000 7f4d6511c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.353+0000 7f4d6511c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.353+0000 7f4d6511c640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:50:48.354 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.353+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d60070660 con 0x7f4d6006c780 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2056873053 0 0) 0x7f4d4c003520 con 0x7f4d60149f50 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 >> v1:172.21.15.50:6790/0 conn(0x7f4d600b1b10 legacy=0x7f4d60152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 >> v1:172.21.15.50:6789/0 conn(0x7f4d6006c780 legacy=0x7f4d600b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 1 monclient: found mon.noname-c 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6006d3e0 con 0x7f4d60149f50 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4d5c001690 con 0x7f4d60149f50 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d68382640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d68382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d6511c640 10 monclient: dump: 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:48.358 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.356+0000 7f4d68382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:48.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d5c0017f0 con 0x7f4d60149f50 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d6511c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d6511c640 1 -- 172.21.15.50:0/3511540694 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4d5c001c00 con 0x7f4d60149f50 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d68382640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d68382640 4 set_mon_vals no callback set 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d6511c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d6511c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.370+0000 7f4d6511c640 10 monclient: dump: 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:48.370 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:48.371 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:48.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.371+0000 7f4d68382640 10 monclient: shutdown 2024-03-25T19:50:48.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.371+0000 7f4d68382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:48.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.371+0000 7f4d68382640 1 -- 172.21.15.50:0/3511540694 >> v1:172.21.15.76:6789/0 conn(0x7f4d60149f50 legacy=0x7f4d600b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.373 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.371+0000 7f4d68382640 1 -- 172.21.15.50:0/3511540694 shutdown_connections 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.371+0000 7f4d68382640 1 -- 172.21.15.50:0/3511540694 >> 172.21.15.50:0/3511540694 conn(0x7f4d600b5d10 msgr2=0x7f4d600b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.371+0000 7f4d68382640 1 -- 172.21.15.50:0/3511540694 shutdown_connections 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.371+0000 7f4d68382640 1 -- 172.21.15.50:0/3511540694 wait complete. 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 10 monclient: monmap: 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:48.374 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 1 librados: starting msgr at 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 1 librados: starting objecter 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 1 Processor -- start 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 1 -- start start 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 1 librados: setting wanted keys 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 1 librados: calling monclient init 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.372+0000 7f4d68382640 10 monclient: init 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 10 monclient(hunting): picked mon.noname-b con 0x7f4d6006c780 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 10 monclient(hunting): picked mon.noname-c con 0x7f4d600b1b10 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 10 monclient(hunting): picked mon.noname-a con 0x7f4d60149f50 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d601dca00 con 0x7f4d60149f50 2024-03-25T19:50:48.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600af330 con 0x7f4d6006c780 2024-03-25T19:50:48.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600accd0 con 0x7f4d600b1b10 2024-03-25T19:50:48.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.373+0000 7f4d68382640 10 monclient(hunting): authenticate will time out at 5423.637453s 2024-03-25T19:50:48.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.384+0000 7f4d6591d640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f4d600b1b10 0x7f4d600ad930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:45296/0 (socket says 172.21.15.50:45296) 2024-03-25T19:50:48.383 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.384+0000 7f4d6591d640 1 -- 172.21.15.50:0/4154924224 learned_addr learned my addr 172.21.15.50:0/4154924224 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.387+0000 7f4d56ffd640 10 client.?.objecter ms_handle_connect 0x7f4d600b1b10 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.387+0000 7f4d56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.390+0000 7f4d56ffd640 10 client.?.objecter ms_handle_connect 0x7f4d6006c780 2024-03-25T19:50:48.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.390+0000 7f4d56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.394 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.395+0000 7f4d56ffd640 10 client.?.objecter ms_handle_connect 0x7f4d60149f50 2024-03-25T19:50:48.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.395+0000 7f4d56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.399+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120746047 0 0) 0x7f4d600accd0 con 0x7f4d600b1b10 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.399+0000 7f4d56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.399+0000 7f4d56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.399+0000 7f4d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.399+0000 7f4d56ffd640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.399+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d44004eb0 con 0x7f4d600b1b10 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.400+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013660897 0 0) 0x7f4d600af330 con 0x7f4d6006c780 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.400+0000 7f4d56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.400+0000 7f4d56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.400+0000 7f4d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.400+0000 7f4d56ffd640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.400+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d600accd0 con 0x7f4d6006c780 2024-03-25T19:50:48.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.403+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310429079 0 0) 0x7f4d601dca00 con 0x7f4d60149f50 2024-03-25T19:50:48.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.403+0000 7f4d56ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.402 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.403+0000 7f4d56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.403+0000 7f4d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.403+0000 7f4d56ffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:50:48.403 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.403+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d600af330 con 0x7f4d60149f50 2024-03-25T19:50:48.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.411+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2061949229 0 0) 0x7f4d600accd0 con 0x7f4d6006c780 2024-03-25T19:50:48.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.411+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d601dca00 con 0x7f4d6006c780 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.411+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4d5c001b40 con 0x7f4d6006c780 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.411+0000 7f4d56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.411+0000 7f4d56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.411+0000 7f4d56ffd640 10 monclient(hunting): dump: 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:48.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.411+0000 7f4d56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:48.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.419+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 458490855 0 0) 0x7f4d600af330 con 0x7f4d60149f50 2024-03-25T19:50:48.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.419+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d600accd0 con 0x7f4d60149f50 2024-03-25T19:50:48.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.419+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4d58001890 con 0x7f4d60149f50 2024-03-25T19:50:48.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.419+0000 7f4d56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.420 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.419+0000 7f4d56ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.419+0000 7f4d56ffd640 10 monclient(hunting): dump: 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1771920331 0 0) 0x7f4d601dca00 con 0x7f4d6006c780 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 >> v1:172.21.15.50:6790/0 conn(0x7f4d600b1b10 legacy=0x7f4d600ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.421 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 >> v1:172.21.15.50:6789/0 conn(0x7f4d60149f50 legacy=0x7f4d601d8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 1 monclient: found mon.b 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600ad410 con 0x7f4d6006c780 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 5 monclient: authenticate success, global_id 4666 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 10 monclient: _renew_subs 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d600ae040 con 0x7f4d6006c780 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:48.422 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 10 monclient: _renew_subs 2024-03-25T19:50:48.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:48.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d600ad160 con 0x7f4d6006c780 2024-03-25T19:50:48.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.420+0000 7f4d68382640 1 librados: init done 2024-03-25T19:50:48.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.421+0000 7f4d68382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.421+0000 7f4d68382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.421+0000 7f4d68382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:48.423 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.421+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d600627b0 con 0x7f4d6006c780 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.706+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d5c001d60 con 0x7f4d6006c780 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.706+0000 7f4d56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.706+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f4d5c004c30 con 0x7f4d6006c780 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.706+0000 7f4d56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.706+0000 7f4d56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.706+0000 7f4d56ffd640 10 monclient: dump: 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:48.706 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.707 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:48.707 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:48.707 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:48.707 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:48.707 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:48.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.706+0000 7f4d6491b640 4 set_mon_vals no callback set 2024-03-25T19:50:48.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.707+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f4d5c01ac00 con 0x7f4d6006c780 2024-03-25T19:50:48.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.707+0000 7f4d6591d640 10 monclient: get_auth_request con 0x7f4d4406df30 auth_method 0 2024-03-25T19:50:48.708 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.708+0000 7f4d56ffd640 10 client.4666.objecter ms_handle_connect 0x7f4d4406df30 2024-03-25T19:50:48.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.713+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7f4d5c054780 con 0x7f4d6006c780 2024-03-25T19:50:48.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.713+0000 7f4d56ffd640 10 client.4666.objecter ms_dispatch 0x7f4d60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:48.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.713+0000 7f4d56ffd640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:48.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.713+0000 7f4d56ffd640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:48.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.713+0000 7f4d56ffd640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T19:50:48.713 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.714+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d5c0500c0 con 0x7f4d6006c780 2024-03-25T19:50:48.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.714+0000 7f4d56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.714 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.714+0000 7f4d56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:48.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.941+0000 7f4d68382640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:48.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.941+0000 7f4d68382640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:48.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.941+0000 7f4d68382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:50:48.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:48.941+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d600b4d00 con 0x7f4d6006c780 2024-03-25T19:50:49.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.034+0000 7f4d56ffd640 1 -- 172.21.15.50:0/4154924224 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7f4d5c0521c0 con 0x7f4d6006c780 2024-03-25T19:50:49.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.034+0000 7f4d56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:49.034 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.034+0000 7f4d56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:49.034 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:49.034 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 10 librados: watch_flush enter 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 10 librados: watch_flush exit 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 10 client.4666.objecter successfully canceled tick 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 >> v1:172.21.15.76:6800/2572556811 conn(0x7f4d4406df30 legacy=0x7f4d440703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 10 monclient: shutdown 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 >> v1:172.21.15.76:6789/0 conn(0x7f4d6006c780 legacy=0x7f4d601dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 shutdown_connections 2024-03-25T19:50:49.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 >> 172.21.15.50:0/4154924224 conn(0x7f4d600b5d10 msgr2=0x7f4d600b4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:49.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 shutdown_connections 2024-03-25T19:50:49.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 1 -- 172.21.15.50:0/4154924224 wait complete. 2024-03-25T19:50:49.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:49.037+0000 7f4d68382640 1 librados: shutdown 2024-03-25T19:50:49.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:55.855 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.968+0000 7feb2dd9a640 10 monclient: get_monmap_and_config 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.968+0000 7feb2dd9a640 10 monclient: build_initial_monmap 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.968+0000 7feb2dd9a640 10 monclient: monmap: 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:50:55.969299+0000 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:50:55.969299+0000 2024-03-25T19:50:55.968 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.969+0000 7feb2dd9a640 1 Processor -- start 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.969+0000 7feb2dd9a640 1 -- start start 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.969+0000 7feb2dd9a640 10 monclient: init 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 10 monclient(hunting): picked mon.noname-b con 0x7feb280b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 10 monclient(hunting): picked mon.noname-c con 0x7feb280b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 10 monclient(hunting): picked mon.noname-a con 0x7feb2814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280af940 con 0x7feb2814be90 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280b0a70 con 0x7feb280b1730 2024-03-25T19:50:55.970 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb280b0d50 con 0x7feb280b4d60 2024-03-25T19:50:55.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:55.971 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.970+0000 7feb2dd9a640 10 monclient(hunting): authenticate will time out at 5431.234077s 2024-03-25T19:50:55.976 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.977+0000 7feb26ffd640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7feb280b1730 0x7feb280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:46092/0 (socket says 172.21.15.50:46092) 2024-03-25T19:50:55.977 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:55.977+0000 7feb26ffd640 1 -- 172.21.15.50:0/1469810489 learned_addr learned my addr 172.21.15.50:0/1469810489 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:56.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.002+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972587648 0 0) 0x7feb280b0d50 con 0x7feb280b4d60 2024-03-25T19:50:56.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.002+0000 7feb267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.002+0000 7feb267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.002+0000 7feb267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.002+0000 7feb267fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:50:56.002 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.002+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb10003520 con 0x7feb280b4d60 2024-03-25T19:50:56.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819282035 0 0) 0x7feb10003520 con 0x7feb280b4d60 2024-03-25T19:50:56.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 >> v1:172.21.15.76:6789/0 conn(0x7feb280b1730 legacy=0x7feb280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 >> v1:172.21.15.50:6789/0 conn(0x7feb2814be90 legacy=0x7feb2814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:56.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 1 monclient: found mon.noname-b 2024-03-25T19:50:56.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb280b1030 con 0x7feb280b4d60 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feb18001690 con 0x7feb280b4d60 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb267fc640 10 monclient: dump: 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.013 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb2dd9a640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.012+0000 7feb2dd9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:56.037 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb180017f0 con 0x7feb280b4d60 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb267fc640 1 -- 172.21.15.50:0/1469810489 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feb18001c00 con 0x7feb280b4d60 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb267fc640 10 monclient: dump: 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.038 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:56.039 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:56.039 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:56.039 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:56.039 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:56.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb2dd9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:56.039 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.038+0000 7feb2dd9a640 4 set_mon_vals no callback set 2024-03-25T19:50:56.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.039+0000 7feb2dd9a640 10 monclient: shutdown 2024-03-25T19:50:56.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.039+0000 7feb2dd9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:56.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.039+0000 7feb2dd9a640 1 -- 172.21.15.50:0/1469810489 >> v1:172.21.15.50:6790/0 conn(0x7feb280b4d60 legacy=0x7feb2813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.039+0000 7feb2dd9a640 1 -- 172.21.15.50:0/1469810489 shutdown_connections 2024-03-25T19:50:56.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.039+0000 7feb2dd9a640 1 -- 172.21.15.50:0/1469810489 >> 172.21.15.50:0/1469810489 conn(0x7feb280617a0 msgr2=0x7feb28061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:56.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.039+0000 7feb2dd9a640 1 -- 172.21.15.50:0/1469810489 shutdown_connections 2024-03-25T19:50:56.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.039+0000 7feb2dd9a640 1 -- 172.21.15.50:0/1469810489 wait complete. 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 10 monclient: build_initial_monmap 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 10 monclient: monmap: 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:50:56.042 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 1 librados: starting msgr at 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 1 librados: starting objecter 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 1 Processor -- start 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 1 -- start start 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 1 librados: setting wanted keys 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 1 librados: calling monclient init 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.040+0000 7feb2dd9a640 10 monclient: init 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 10 monclient(hunting): picked mon.noname-b con 0x7feb280b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 10 monclient(hunting): picked mon.noname-c con 0x7feb280b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:50:56.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 10 monclient(hunting): picked mon.noname-a con 0x7feb2814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:50:56.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb28141bf0 con 0x7feb2814be90 2024-03-25T19:50:56.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb28141ed0 con 0x7feb280b1730 2024-03-25T19:50:56.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb281421b0 con 0x7feb280b4d60 2024-03-25T19:50:56.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:56.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.041+0000 7feb2dd9a640 10 monclient(hunting): authenticate will time out at 5431.305426s 2024-03-25T19:50:56.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.051+0000 7feb26ffd640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7feb280b4d60 0x7feb28133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:49716/0 (socket says 172.21.15.50:49716) 2024-03-25T19:50:56.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.051+0000 7feb26ffd640 1 -- 172.21.15.50:0/3425824192 learned_addr learned my addr 172.21.15.50:0/3425824192 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:50:56.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.053+0000 7feb0ffff640 10 client.?.objecter ms_handle_connect 0x7feb280b4d60 2024-03-25T19:50:56.052 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.053+0000 7feb0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.057+0000 7feb0ffff640 10 client.?.objecter ms_handle_connect 0x7feb280b1730 2024-03-25T19:50:56.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.057+0000 7feb0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.064+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283447741 0 0) 0x7feb281421b0 con 0x7feb280b4d60 2024-03-25T19:50:56.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.064+0000 7feb0ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.064+0000 7feb0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.064+0000 7feb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.064+0000 7feb0ffff640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:50:56.064 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.064+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb14004eb0 con 0x7feb280b4d60 2024-03-25T19:50:56.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.071+0000 7feb0ffff640 10 client.?.objecter ms_handle_connect 0x7feb2814be90 2024-03-25T19:50:56.071 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.071+0000 7feb0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.073+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441209441 0 0) 0x7feb28141ed0 con 0x7feb280b1730 2024-03-25T19:50:56.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.073+0000 7feb0ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.073+0000 7feb0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.073+0000 7feb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.073+0000 7feb0ffff640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:50:56.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.073+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb281421b0 con 0x7feb280b1730 2024-03-25T19:50:56.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.092+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983066882 0 0) 0x7feb28141bf0 con 0x7feb2814be90 2024-03-25T19:50:56.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.092+0000 7feb0ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.092+0000 7feb0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.092+0000 7feb0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.092+0000 7feb0ffff640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:50:56.092 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.092+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb28141ed0 con 0x7feb2814be90 2024-03-25T19:50:56.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.093+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 280499358 0 0) 0x7feb14004eb0 con 0x7feb280b4d60 2024-03-25T19:50:56.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.093+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb28141bf0 con 0x7feb280b4d60 2024-03-25T19:50:56.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.093+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feb1c0018b0 con 0x7feb280b4d60 2024-03-25T19:50:56.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.093+0000 7feb0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.093+0000 7feb0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.093+0000 7feb0ffff640 10 monclient(hunting): dump: 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:56.094 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.093+0000 7feb0ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.106+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1521262251 0 0) 0x7feb281421b0 con 0x7feb280b1730 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.106+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb14004eb0 con 0x7feb280b1730 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.106+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feb18001b80 con 0x7feb280b1730 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.106+0000 7feb0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.106+0000 7feb0ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.106+0000 7feb0ffff640 10 monclient(hunting): dump: 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.106 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.107 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:56.107 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:56.107 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:56.107 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:56.107 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.117+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3168740705 0 0) 0x7feb28141bf0 con 0x7feb280b4d60 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 >> v1:172.21.15.76:6789/0 conn(0x7feb280b1730 legacy=0x7feb281425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 >> v1:172.21.15.50:6789/0 conn(0x7feb2814be90 legacy=0x7feb281341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 1 monclient: found mon.c 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb28140f00 con 0x7feb280b4d60 2024-03-25T19:50:56.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 10 monclient: _finish_auth 0 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 5 monclient: authenticate success, global_id 4661 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 10 monclient: _renew_subs 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb281406f0 con 0x7feb280b4d60 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 10 monclient: _renew_subs 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb28140380 con 0x7feb280b4d60 2024-03-25T19:50:56.119 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.118+0000 7feb2dd9a640 1 librados: init done 2024-03-25T19:50:56.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.119+0000 7feb2dd9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.119+0000 7feb2dd9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.119+0000 7feb2dd9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:56.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.119+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb281401f0 con 0x7feb280b4d60 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.426+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb1c0047e0 con 0x7feb280b4d60 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.426+0000 7feb0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.426+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7feb1c004c30 con 0x7feb280b4d60 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.426+0000 7feb0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.426+0000 7feb0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.426+0000 7feb0ffff640 10 monclient: dump: 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.426 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.426+0000 7feb25ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.427+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7feb1c004e50 con 0x7feb280b4d60 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.427+0000 7feb277fe640 10 monclient: get_auth_request con 0x7feb1406df10 auth_method 0 2024-03-25T19:50:56.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.428+0000 7feb0ffff640 10 client.4661.objecter ms_handle_connect 0x7feb1406df10 2024-03-25T19:50:56.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.432+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1569349235 0 0) 0x7feb1c0547f0 con 0x7feb280b4d60 2024-03-25T19:50:56.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.432+0000 7feb0ffff640 10 client.4661.objecter ms_dispatch 0x7feb28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:56.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.432+0000 7feb0ffff640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:56.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.432+0000 7feb0ffff640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:56.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.432+0000 7feb0ffff640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-25T19:50:56.432 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.433+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb1c050300 con 0x7feb280b4d60 2024-03-25T19:50:56.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.433+0000 7feb0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.433 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.433+0000 7feb0ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:56.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.687+0000 7feb2dd9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.687+0000 7feb2dd9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.687+0000 7feb2dd9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:50:56.687 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.687+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb28140880 con 0x7feb280b4d60 2024-03-25T19:50:56.728 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.729+0000 7feb0ffff640 1 -- 172.21.15.50:0/3425824192 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2731738098) 0x7feb1c052230 con 0x7feb280b4d60 2024-03-25T19:50:56.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.729+0000 7feb0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.729 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.729+0000 7feb0ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:56.729 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:50:56.729 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":10,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:41:02.314093+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:02.314093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.731+0000 7feb2dd9a640 10 librados: watch_flush enter 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 10 librados: watch_flush exit 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 10 client.4661.objecter successfully canceled tick 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 >> v1:172.21.15.76:6800/2572556811 conn(0x7feb1406df10 legacy=0x7feb140703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 10 monclient: shutdown 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 >> v1:172.21.15.50:6790/0 conn(0x7feb280b4d60 legacy=0x7feb28133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 shutdown_connections 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 >> 172.21.15.50:0/3425824192 conn(0x7feb280617a0 msgr2=0x7feb2813c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 shutdown_connections 2024-03-25T19:50:56.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 1 -- 172.21.15.50:0/3425824192 wait complete. 2024-03-25T19:50:56.733 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:50:56.732+0000 7feb2dd9a640 1 librados: shutdown 2024-03-25T19:50:56.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:03.551 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.672+0000 7fb8be7af640 10 monclient: get_monmap_and_config 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.672+0000 7fb8be7af640 10 monclient: build_initial_monmap 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.672+0000 7fb8be7af640 10 monclient: monmap: 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:03.673794+0000 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:03.673794+0000 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.673+0000 7fb8be7af640 1 Processor -- start 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 1 -- start start 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient: init 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient(hunting): picked mon.noname-a con 0x7fb8b80a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient(hunting): picked mon.noname-b con 0x7fb8b813f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:03.674 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient(hunting): picked mon.noname-c con 0x7fb8b81430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:03.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8142520 con 0x7fb8b80a98f0 2024-03-25T19:51:03.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80b28a0 con 0x7fb8b81430c0 2024-03-25T19:51:03.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80b2a30 con 0x7fb8b813f710 2024-03-25T19:51:03.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:03.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.674+0000 7fb8be7af640 10 monclient(hunting): authenticate will time out at 5438.938513s 2024-03-25T19:51:03.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.689+0000 7fb8b77fe640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fb8b813f710 0x7fb8b81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:43142/0 (socket says 172.21.15.50:43142) 2024-03-25T19:51:03.689 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.689+0000 7fb8b77fe640 1 -- 172.21.15.50:0/2223497081 learned_addr learned my addr 172.21.15.50:0/2223497081 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:03.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.698+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579162545 0 0) 0x7fb8b80b2a30 con 0x7fb8b813f710 2024-03-25T19:51:03.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.698+0000 7fb8b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.698+0000 7fb8b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.698+0000 7fb8b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.698+0000 7fb8b6ffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:51:03.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.698+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a0003520 con 0x7fb8b813f710 2024-03-25T19:51:03.700 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.701+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568664316 0 0) 0x7fb8b8142520 con 0x7fb8b80a98f0 2024-03-25T19:51:03.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.701+0000 7fb8b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.701+0000 7fb8b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.701+0000 7fb8b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.701+0000 7fb8b6ffd640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:51:03.701 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.701+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b80b2a30 con 0x7fb8b80a98f0 2024-03-25T19:51:03.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.706+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502600191 0 0) 0x7fb8b80b28a0 con 0x7fb8b81430c0 2024-03-25T19:51:03.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.706+0000 7fb8b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.706+0000 7fb8b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.706+0000 7fb8b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.706+0000 7fb8b6ffd640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:51:03.706 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.706+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b8142520 con 0x7fb8b81430c0 2024-03-25T19:51:03.724 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 354300636 0 0) 0x7fb8b80b2a30 con 0x7fb8b80a98f0 2024-03-25T19:51:03.724 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 >> v1:172.21.15.50:6790/0 conn(0x7fb8b813f710 legacy=0x7fb8b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.724 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 >> v1:172.21.15.76:6789/0 conn(0x7fb8b81430c0 legacy=0x7fb8b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.724 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80b44c0 con 0x7fb8b80a98f0 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb8a8001670 con 0x7fb8b80a98f0 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8b6ffd640 10 monclient: dump: 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.725 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8be7af640 5 monclient: authenticate success, global_id 4698 2024-03-25T19:51:03.726 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.724+0000 7fb8be7af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:03.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a80017d0 con 0x7fb8b80a98f0 2024-03-25T19:51:03.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:03.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8b6ffd640 1 -- 172.21.15.50:0/2223497081 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb8a8001be0 con 0x7fb8b80a98f0 2024-03-25T19:51:03.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8be7af640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:03.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8be7af640 4 set_mon_vals no callback set 2024-03-25T19:51:03.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.753 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.753+0000 7fb8b6ffd640 10 monclient: dump: 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:03.754 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:03.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.754+0000 7fb8be7af640 10 monclient: shutdown 2024-03-25T19:51:03.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.754+0000 7fb8be7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:03.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.754+0000 7fb8be7af640 1 -- 172.21.15.50:0/2223497081 >> v1:172.21.15.50:6789/0 conn(0x7fb8b80a98f0 legacy=0x7fb8b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.755 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.754+0000 7fb8be7af640 1 -- 172.21.15.50:0/2223497081 shutdown_connections 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.754+0000 7fb8be7af640 1 -- 172.21.15.50:0/2223497081 >> 172.21.15.50:0/2223497081 conn(0x7fb8b80b7ea0 msgr2=0x7fb8b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.754+0000 7fb8be7af640 1 -- 172.21.15.50:0/2223497081 shutdown_connections 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.754+0000 7fb8be7af640 1 -- 172.21.15.50:0/2223497081 wait complete. 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.755+0000 7fb8be7af640 10 monclient: build_initial_monmap 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.755+0000 7fb8be7af640 10 monclient: monmap: 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:03.756 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:03.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.756+0000 7fb8be7af640 1 librados: starting msgr at 2024-03-25T19:51:03.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.756+0000 7fb8be7af640 1 librados: starting objecter 2024-03-25T19:51:03.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.756+0000 7fb8be7af640 1 Processor -- start 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.756+0000 7fb8be7af640 1 -- start start 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.756+0000 7fb8be7af640 1 librados: setting wanted keys 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.756+0000 7fb8be7af640 1 librados: calling monclient init 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.756+0000 7fb8be7af640 10 monclient: init 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 10 monclient(hunting): picked mon.noname-c con 0x7fb8b80a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 10 monclient(hunting): picked mon.noname-a con 0x7fb8b813f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 10 monclient(hunting): picked mon.noname-b con 0x7fb8b81430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:03.758 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8138f10 con 0x7fb8b813f710 2024-03-25T19:51:03.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b8139970 con 0x7fb8b81430c0 2024-03-25T19:51:03.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80abb50 con 0x7fb8b80a98f0 2024-03-25T19:51:03.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:03.759 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.757+0000 7fb8be7af640 10 monclient(hunting): authenticate will time out at 5439.021025s 2024-03-25T19:51:03.766 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.766+0000 7fb8b7fff640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fb8b80a98f0 0x7fb8b8136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:43194/0 (socket says 172.21.15.50:43194) 2024-03-25T19:51:03.766 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.766+0000 7fb8b7fff640 1 -- 172.21.15.50:0/2959130785 learned_addr learned my addr 172.21.15.50:0/2959130785 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:03.767 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.768+0000 7fb8b4ff9640 10 client.?.objecter ms_handle_connect 0x7fb8b80a98f0 2024-03-25T19:51:03.767 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.768+0000 7fb8b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.772 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.772+0000 7fb8b4ff9640 10 client.?.objecter ms_handle_connect 0x7fb8b813f710 2024-03-25T19:51:03.772 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.772+0000 7fb8b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.773 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.774+0000 7fb8b4ff9640 10 client.?.objecter ms_handle_connect 0x7fb8b81430c0 2024-03-25T19:51:03.774 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.774+0000 7fb8b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.787+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812702409 0 0) 0x7fb8b80abb50 con 0x7fb8b80a98f0 2024-03-25T19:51:03.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.787+0000 7fb8b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.787+0000 7fb8b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.787+0000 7fb8b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.787+0000 7fb8b4ff9640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:51:03.787 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.787+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4004eb0 con 0x7fb8b80a98f0 2024-03-25T19:51:03.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097949669 0 0) 0x7fb8b8139970 con 0x7fb8b81430c0 2024-03-25T19:51:03.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:51:03.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b80abb50 con 0x7fb8b81430c0 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015926631 0 0) 0x7fb8a4004eb0 con 0x7fb8b80a98f0 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b8139970 con 0x7fb8b80a98f0 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb8a8001670 con 0x7fb8b80a98f0 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.801 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.800+0000 7fb8b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.801+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780384373 0 0) 0x7fb8b8138f10 con 0x7fb8b813f710 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.801+0000 7fb8b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.801+0000 7fb8b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.801+0000 7fb8b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.801+0000 7fb8b4ff9640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:51:03.802 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.801+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4004eb0 con 0x7fb8b813f710 2024-03-25T19:51:03.822 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.823+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1623283548 0 0) 0x7fb8b80abb50 con 0x7fb8b81430c0 2024-03-25T19:51:03.822 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.823+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8b8138f10 con 0x7fb8b81430c0 2024-03-25T19:51:03.822 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.823+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb8b0001b50 con 0x7fb8b81430c0 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.823+0000 7fb8b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.823+0000 7fb8b4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.823+0000 7fb8b4ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:03.823 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3039288679 0 0) 0x7fb8b8139970 con 0x7fb8b80a98f0 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 >> v1:172.21.15.76:6789/0 conn(0x7fb8b81430c0 legacy=0x7fb8b8138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 >> v1:172.21.15.50:6789/0 conn(0x7fb8b813f710 legacy=0x7fb8b80ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 1 monclient: found mon.c 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80abe30 con 0x7fb8b80a98f0 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8be7af640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:51:03.828 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8be7af640 10 monclient: _renew_subs 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8be7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.827+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b8137b40 con 0x7fb8b80a98f0 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.828+0000 7fb8be7af640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.828+0000 7fb8be7af640 10 monclient: _renew_subs 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.828+0000 7fb8be7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.828+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b8138c10 con 0x7fb8b80a98f0 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.828+0000 7fb8be7af640 1 librados: init done 2024-03-25T19:51:03.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.829+0000 7fb8be7af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:03.830 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.829+0000 7fb8be7af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:03.830 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.829+0000 7fb8be7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:03.830 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:03.829+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b80627b0 con 0x7fb8b80a98f0 2024-03-25T19:51:04.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.154+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8a80039a0 con 0x7fb8b80a98f0 2024-03-25T19:51:04.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.154+0000 7fb8b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:04.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.154+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fb8a8001ca0 con 0x7fb8b80a98f0 2024-03-25T19:51:04.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.154+0000 7fb8b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:04.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.154+0000 7fb8b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:04.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.154+0000 7fb8b4ff9640 10 monclient: dump: 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:04.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.154+0000 7fb8b67fc640 4 set_mon_vals no callback set 2024-03-25T19:51:04.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.155+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fb8a8003eb0 con 0x7fb8b80a98f0 2024-03-25T19:51:04.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.156+0000 7fb8b77fe640 10 monclient: get_auth_request con 0x7fb8a406dfb0 auth_method 0 2024-03-25T19:51:04.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.156+0000 7fb8b4ff9640 10 client.4667.objecter ms_handle_connect 0x7fb8a406dfb0 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7fb8a8054950 con 0x7fb8b80a98f0 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 10 client.4667.objecter ms_dispatch 0x7fb8b8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8a8050460 con 0x7fb8b80a98f0 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:04.161 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.161+0000 7fb8b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:04.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.415+0000 7fb8be7af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.415+0000 7fb8be7af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.415+0000 7fb8be7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:04.415 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.415+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8b813db40 con 0x7fb8b80a98f0 2024-03-25T19:51:04.458 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.458+0000 7fb8b4ff9640 1 -- 172.21.15.50:0/2959130785 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7fb8a8052390 con 0x7fb8b80a98f0 2024-03-25T19:51:04.458 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.458+0000 7fb8b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.458 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.458+0000 7fb8b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:04.458 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:04.458 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 10 librados: watch_flush enter 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 10 librados: watch_flush exit 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 10 client.4667.objecter successfully canceled tick 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 >> v1:172.21.15.76:6800/2572556811 conn(0x7fb8a406dfb0 legacy=0x7fb8a4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 10 monclient: shutdown 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.460+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 >> v1:172.21.15.50:6790/0 conn(0x7fb8b80a98f0 legacy=0x7fb8b8136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.461+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 shutdown_connections 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.461+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 >> 172.21.15.50:0/2959130785 conn(0x7fb8b80b7ea0 msgr2=0x7fb8b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.461+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 shutdown_connections 2024-03-25T19:51:04.461 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.461+0000 7fb8be7af640 1 -- 172.21.15.50:0/2959130785 wait complete. 2024-03-25T19:51:04.462 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:04.461+0000 7fb8be7af640 1 librados: shutdown 2024-03-25T19:51:04.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:06.201 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:26.707 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.257 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:26.763 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.259 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:26.765 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.291 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:26.797 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.315 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:26.821 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.321 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:26.792 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.326 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:26.796 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.349 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:26.855 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:06.353 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:26.824 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:11.279 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:11.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.395+0000 7f9cae6dd640 10 monclient: get_monmap_and_config 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.395+0000 7f9cae6dd640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.395+0000 7f9cae6dd640 10 monclient: monmap: 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:11.396745+0000 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:11.396745+0000 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:11.396 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.396+0000 7f9cae6dd640 1 Processor -- start 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.396+0000 7f9cae6dd640 1 -- start start 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.396+0000 7f9cae6dd640 10 monclient: init 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca80b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca80b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca814be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca80af940 con 0x7f9ca80b4d60 2024-03-25T19:51:11.398 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca80b0a70 con 0x7f9ca80b1730 2024-03-25T19:51:11.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca80b0d50 con 0x7f9ca814be90 2024-03-25T19:51:11.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.399 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.397+0000 7f9cae6dd640 10 monclient(hunting): authenticate will time out at 5446.661385s 2024-03-25T19:51:11.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.411+0000 7f9ca7fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f9ca80b4d60 0x7f9ca813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:55124/0 (socket says 172.21.15.50:55124) 2024-03-25T19:51:11.411 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.412+0000 7f9ca7fff640 1 -- 172.21.15.50:0/1726926569 learned_addr learned my addr 172.21.15.50:0/1726926569 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:11.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.426+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245508786 0 0) 0x7f9ca80af940 con 0x7f9ca80b4d60 2024-03-25T19:51:11.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.426+0000 7f9ca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.425 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.426+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.426+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.426+0000 7f9ca6ffd640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:51:11.426 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.426+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c90003520 con 0x7f9ca80b4d60 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.427+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644179038 0 0) 0x7f9ca80b0d50 con 0x7f9ca814be90 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.427+0000 7f9ca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.427+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.427+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.427+0000 7f9ca6ffd640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.427+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca80af940 con 0x7f9ca814be90 2024-03-25T19:51:11.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.434+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699598538 0 0) 0x7f9ca80b0a70 con 0x7f9ca80b1730 2024-03-25T19:51:11.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.434+0000 7f9ca6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.434+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.434+0000 7f9ca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.434+0000 7f9ca6ffd640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:51:11.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.434+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca80b0d50 con 0x7f9ca80b1730 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 386793572 0 0) 0x7f9c90003520 con 0x7f9ca80b4d60 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 >> v1:172.21.15.50:6790/0 conn(0x7f9ca814be90 legacy=0x7f9ca814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 >> v1:172.21.15.76:6789/0 conn(0x7f9ca80b1730 legacy=0x7f9ca80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 1 monclient: found mon.noname-a 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80b1030 con 0x7f9ca80b4d60 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9c98001690 con 0x7f9ca80b4d60 2024-03-25T19:51:11.443 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9cae6dd640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9ca6ffd640 10 monclient: dump: 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.444 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.443+0000 7f9cae6dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c980017f0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9ca6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9ca6ffd640 1 -- 172.21.15.50:0/1726926569 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9c98001c00 con 0x7f9ca80b4d60 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9cae6dd640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9cae6dd640 4 set_mon_vals no callback set 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9ca6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.452+0000 7f9ca6ffd640 10 monclient: dump: 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.452 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:11.453 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:11.453 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:11.453 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:11.453 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.453+0000 7f9cae6dd640 10 monclient: shutdown 2024-03-25T19:51:11.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.453+0000 7f9cae6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:11.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.453+0000 7f9cae6dd640 1 -- 172.21.15.50:0/1726926569 >> v1:172.21.15.50:6789/0 conn(0x7f9ca80b4d60 legacy=0x7f9ca813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.453+0000 7f9cae6dd640 1 -- 172.21.15.50:0/1726926569 shutdown_connections 2024-03-25T19:51:11.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.453+0000 7f9cae6dd640 1 -- 172.21.15.50:0/1726926569 >> 172.21.15.50:0/1726926569 conn(0x7f9ca80617a0 msgr2=0x7f9ca8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:11.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.453+0000 7f9cae6dd640 1 -- 172.21.15.50:0/1726926569 shutdown_connections 2024-03-25T19:51:11.454 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.453+0000 7f9cae6dd640 1 -- 172.21.15.50:0/1726926569 wait complete. 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 10 monclient: monmap: 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.455 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 1 librados: starting msgr at 2024-03-25T19:51:11.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 1 librados: starting objecter 2024-03-25T19:51:11.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 1 Processor -- start 2024-03-25T19:51:11.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 1 -- start start 2024-03-25T19:51:11.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 1 librados: setting wanted keys 2024-03-25T19:51:11.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 1 librados: calling monclient init 2024-03-25T19:51:11.456 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.454+0000 7f9cae6dd640 10 monclient: init 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 10 monclient(hunting): picked mon.noname-a con 0x7f9ca80b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 10 monclient(hunting): picked mon.noname-c con 0x7f9ca80b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 10 monclient(hunting): picked mon.noname-b con 0x7f9ca814be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca8136540 con 0x7f9ca80b1730 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca8138360 con 0x7f9ca814be90 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca8137300 con 0x7f9ca80b4d60 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.457 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.455+0000 7f9cae6dd640 10 monclient(hunting): authenticate will time out at 5446.719316s 2024-03-25T19:51:11.468 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.468+0000 7f9cacc7a640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9ca814be90 0x7f9ca812fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:55522/0 (socket says 172.21.15.50:55522) 2024-03-25T19:51:11.468 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.468+0000 7f9cacc7a640 1 -- 172.21.15.50:0/393424182 learned_addr learned my addr 172.21.15.50:0/393424182 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:11.468 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.469+0000 7f9ca4ff9640 10 client.?.objecter ms_handle_connect 0x7f9ca814be90 2024-03-25T19:51:11.468 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.469+0000 7f9ca4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.474 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.474+0000 7f9ca4ff9640 10 client.?.objecter ms_handle_connect 0x7f9ca80b4d60 2024-03-25T19:51:11.474 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.474+0000 7f9ca4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.479+0000 7f9ca4ff9640 10 client.?.objecter ms_handle_connect 0x7f9ca80b1730 2024-03-25T19:51:11.478 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.479+0000 7f9ca4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.485 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.485+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357862833 0 0) 0x7f9ca8137300 con 0x7f9ca80b4d60 2024-03-25T19:51:11.485 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.485+0000 7f9ca4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.485 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.485+0000 7f9ca4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.485 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.485+0000 7f9ca4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.485 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.485+0000 7f9ca4ff9640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:51:11.485 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.485+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94004eb0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.488+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886009388 0 0) 0x7f9ca8138360 con 0x7f9ca814be90 2024-03-25T19:51:11.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.488+0000 7f9ca4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.488+0000 7f9ca4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.488+0000 7f9ca4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.488+0000 7f9ca4ff9640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:51:11.488 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.488+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca8137300 con 0x7f9ca814be90 2024-03-25T19:51:11.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.506+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520912466 0 0) 0x7f9ca8136540 con 0x7f9ca80b1730 2024-03-25T19:51:11.506 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.506+0000 7f9ca4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.506+0000 7f9ca4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.506+0000 7f9ca4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.506+0000 7f9ca4ff9640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.506+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca8138360 con 0x7f9ca80b1730 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.507+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715509321 0 0) 0x7f9c94004eb0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.507+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ca8136540 con 0x7f9ca80b4d60 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.507+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9c9c0018f0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.507+0000 7f9ca4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.507+0000 7f9ca4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.507+0000 7f9ca4ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.507 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:11.508 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:11.508 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:11.508 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:11.508 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.508 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.507+0000 7f9ca4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:11.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.509+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2204912610 0 0) 0x7f9ca8137300 con 0x7f9ca814be90 2024-03-25T19:51:11.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.510+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c94004eb0 con 0x7f9ca814be90 2024-03-25T19:51:11.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.510+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9ca0001c10 con 0x7f9ca814be90 2024-03-25T19:51:11.509 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.510+0000 7f9ca4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.510+0000 7f9ca4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.510+0000 7f9ca4ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:11.510 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.524+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2768425353 0 0) 0x7f9ca8138360 con 0x7f9ca80b1730 2024-03-25T19:51:11.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.524+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ca8137300 con 0x7f9ca80b1730 2024-03-25T19:51:11.523 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.524+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9c98001b00 con 0x7f9ca80b1730 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.524+0000 7f9ca4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.524+0000 7f9ca4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.524+0000 7f9ca4ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:11.524 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2487138088 0 0) 0x7f9ca8136540 con 0x7f9ca80b4d60 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 >> v1:172.21.15.76:6789/0 conn(0x7f9ca814be90 legacy=0x7f9ca812fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 >> v1:172.21.15.50:6789/0 conn(0x7f9ca80b1730 legacy=0x7f9ca8136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 1 monclient: found mon.c 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca81d8450 con 0x7f9ca80b4d60 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.526 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9ca4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 5 monclient: authenticate success, global_id 4673 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 10 monclient: _renew_subs 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca8143900 con 0x7f9ca80b4d60 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 10 monclient: _renew_subs 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca81361c0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.525+0000 7f9cae6dd640 1 librados: init done 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.526+0000 7f9cae6dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.526+0000 7f9cae6dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.526+0000 7f9cae6dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:11.527 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.526+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca81401f0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.845+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c9c003610 con 0x7f9ca80b4d60 2024-03-25T19:51:11.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.845+0000 7f9ca4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:11.845 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.845+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f9c9c003920 con 0x7f9ca80b4d60 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.845+0000 7f9ca67fc640 4 set_mon_vals no callback set 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.845+0000 7f9ca4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.845+0000 7f9ca4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.845+0000 7f9ca4ff9640 10 monclient: dump: 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:11.846 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:11.847 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:11.847 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.846+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f9c9c0198f0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.847 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.847+0000 7f9ca7fff640 10 monclient: get_auth_request con 0x7f9c9406e0f0 auth_method 0 2024-03-25T19:51:11.847 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.847+0000 7f9ca4ff9640 10 client.4673.objecter ms_handle_connect 0x7f9c9406e0f0 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7f9c9c0535d0 con 0x7f9ca80b4d60 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 10 client.4673.objecter ms_dispatch 0x7f9ca8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c9c04ef10 con 0x7f9ca80b4d60 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.852 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:11.852+0000 7f9ca4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.091+0000 7f9cae6dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.091+0000 7f9cae6dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.091+0000 7f9cae6dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:51:12.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.091+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca80627b0 con 0x7f9ca80b4d60 2024-03-25T19:51:12.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.125+0000 7f9ca4ff9640 1 -- 172.21.15.50:0/393424182 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7f9c9c051010 con 0x7f9ca80b4d60 2024-03-25T19:51:12.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.125+0000 7f9ca4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:12.125 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.125+0000 7f9ca4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:12.125 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:12.125 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:12.130 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 10 librados: watch_flush enter 2024-03-25T19:51:12.130 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 10 librados: watch_flush exit 2024-03-25T19:51:12.130 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-25T19:51:12.130 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 10 client.4673.objecter successfully canceled tick 2024-03-25T19:51:12.130 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 >> v1:172.21.15.76:6800/2572556811 conn(0x7f9c9406e0f0 legacy=0x7f9c940705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:12.130 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 10 monclient: shutdown 2024-03-25T19:51:12.130 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:12.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 >> v1:172.21.15.50:6790/0 conn(0x7f9ca80b4d60 legacy=0x7f9ca81d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:12.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 shutdown_connections 2024-03-25T19:51:12.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 >> 172.21.15.50:0/393424182 conn(0x7f9ca80617a0 msgr2=0x7f9ca80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:12.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 shutdown_connections 2024-03-25T19:51:12.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.128+0000 7f9cae6dd640 1 -- 172.21.15.50:0/393424182 wait complete. 2024-03-25T19:51:12.131 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:12.129+0000 7f9cae6dd640 1 librados: shutdown 2024-03-25T19:51:12.139 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:18.945 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.047+0000 7f269ea0e640 10 monclient: get_monmap_and_config 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.047+0000 7f269ea0e640 10 monclient: build_initial_monmap 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.047+0000 7f269ea0e640 10 monclient: monmap: 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:19.049494+0000 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:19.049494+0000 2024-03-25T19:51:19.048 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:19.049 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:19.049 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:19.049 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:19.049 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:19.049 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:19.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.048+0000 7f269ea0e640 1 Processor -- start 2024-03-25T19:51:19.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.048+0000 7f269ea0e640 1 -- start start 2024-03-25T19:51:19.050 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.048+0000 7f269ea0e640 10 monclient: init 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 10 monclient(hunting): picked mon.noname-b con 0x7f26980b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 10 monclient(hunting): picked mon.noname-c con 0x7f26980b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 10 monclient(hunting): picked mon.noname-a con 0x7f269814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26980af940 con 0x7f269814be90 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26980b0a70 con 0x7f26980b1730 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26980b0d50 con 0x7f26980b4d60 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f269ea0e640 10 monclient(hunting): authenticate will time out at 5454.313804s 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f26977fe640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f26980b1730 0x7f26980b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:40776/0 (socket says 172.21.15.50:40776) 2024-03-25T19:51:19.051 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.049+0000 7f26977fe640 1 -- 172.21.15.50:0/3078565496 learned_addr learned my addr 172.21.15.50:0/3078565496 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:19.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.090+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861591199 0 0) 0x7f26980af940 con 0x7f269814be90 2024-03-25T19:51:19.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.090+0000 7f2696ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.090+0000 7f2696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.090+0000 7f2696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.090+0000 7f2696ffd640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:51:19.091 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.090+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2684003520 con 0x7f269814be90 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242451444 0 0) 0x7f2684003520 con 0x7f269814be90 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 >> v1:172.21.15.50:6790/0 conn(0x7f26980b4d60 legacy=0x7f269813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 >> v1:172.21.15.76:6789/0 conn(0x7f26980b1730 legacy=0x7f26980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 1 monclient: found mon.noname-a 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980b1030 con 0x7f269814be90 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2690001a10 con 0x7f269814be90 2024-03-25T19:51:19.104 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f2696ffd640 10 monclient: dump: 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f269ea0e640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:51:19.105 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.103+0000 7f269ea0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:19.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.116+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2690001b70 con 0x7f269814be90 2024-03-25T19:51:19.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.116+0000 7f2696ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:19.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.116+0000 7f2696ffd640 1 -- 172.21.15.50:0/3078565496 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2690001fc0 con 0x7f269814be90 2024-03-25T19:51:19.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.116+0000 7f269ea0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:19.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.117+0000 7f269ea0e640 4 set_mon_vals no callback set 2024-03-25T19:51:19.117 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.117+0000 7f2696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.117+0000 7f2696ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.117+0000 7f2696ffd640 10 monclient: dump: 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:19.118 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:19.119 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:19.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.117+0000 7f269ea0e640 10 monclient: shutdown 2024-03-25T19:51:19.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.117+0000 7f269ea0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:19.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.117+0000 7f269ea0e640 1 -- 172.21.15.50:0/3078565496 >> v1:172.21.15.50:6789/0 conn(0x7f269814be90 legacy=0x7f269814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.118+0000 7f269ea0e640 1 -- 172.21.15.50:0/3078565496 shutdown_connections 2024-03-25T19:51:19.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.118+0000 7f269ea0e640 1 -- 172.21.15.50:0/3078565496 >> 172.21.15.50:0/3078565496 conn(0x7f26980617a0 msgr2=0x7f2698061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:19.120 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.118+0000 7f269ea0e640 1 -- 172.21.15.50:0/3078565496 shutdown_connections 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.118+0000 7f269ea0e640 1 -- 172.21.15.50:0/3078565496 wait complete. 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.118+0000 7f269ea0e640 10 monclient: build_initial_monmap 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.118+0000 7f269ea0e640 10 monclient: monmap: 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:19.121 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.119+0000 7f269ea0e640 1 librados: starting msgr at 2024-03-25T19:51:19.122 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.119+0000 7f269ea0e640 1 librados: starting objecter 2024-03-25T19:51:19.122 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.119+0000 7f269ea0e640 1 Processor -- start 2024-03-25T19:51:19.122 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.119+0000 7f269ea0e640 1 -- start start 2024-03-25T19:51:19.122 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.119+0000 7f269ea0e640 1 librados: setting wanted keys 2024-03-25T19:51:19.122 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.119+0000 7f269ea0e640 1 librados: calling monclient init 2024-03-25T19:51:19.122 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.119+0000 7f269ea0e640 10 monclient: init 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 10 monclient(hunting): picked mon.noname-b con 0x7f26980b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 10 monclient(hunting): picked mon.noname-c con 0x7f26980b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 10 monclient(hunting): picked mon.noname-a con 0x7f269814be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26981405b0 con 0x7f269814be90 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2698135650 con 0x7f26980b1730 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26981357e0 con 0x7f26980b4d60 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:19.123 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.120+0000 7f269ea0e640 10 monclient(hunting): authenticate will time out at 5454.385488s 2024-03-25T19:51:19.140 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.140+0000 7f2697fff640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f26980b1730 0x7f2698134f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:40820/0 (socket says 172.21.15.50:40820) 2024-03-25T19:51:19.140 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.140+0000 7f2697fff640 1 -- 172.21.15.50:0/2607807550 learned_addr learned my addr 172.21.15.50:0/2607807550 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:19.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.140+0000 7f2694ff9640 10 client.?.objecter ms_handle_connect 0x7f26980b1730 2024-03-25T19:51:19.141 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.140+0000 7f2694ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.147 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.146+0000 7f2694ff9640 10 client.?.objecter ms_handle_connect 0x7f26980b4d60 2024-03-25T19:51:19.147 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.146+0000 7f2694ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.149+0000 7f2694ff9640 10 client.?.objecter ms_handle_connect 0x7f269814be90 2024-03-25T19:51:19.149 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.149+0000 7f2694ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:19.164 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.163+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934709302 0 0) 0x7f26981405b0 con 0x7f269814be90 2024-03-25T19:51:19.164 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.163+0000 7f2694ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.164 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.163+0000 7f2694ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.164 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.163+0000 7f2694ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.164 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.163+0000 7f2694ff9640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:51:19.164 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.163+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2678004eb0 con 0x7f269814be90 2024-03-25T19:51:19.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.225+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 404279377 0 0) 0x7f2678004eb0 con 0x7f269814be90 2024-03-25T19:51:19.226 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.225+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26981405b0 con 0x7f269814be90 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.225+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2690001f40 con 0x7f269814be90 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.225+0000 7f2694ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.225+0000 7f2694ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.225+0000 7f2694ff9640 10 monclient(hunting): dump: 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:19.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.225+0000 7f2694ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:19.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3012925335 0 0) 0x7f26981405b0 con 0x7f269814be90 2024-03-25T19:51:19.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 >> v1:172.21.15.50:6790/0 conn(0x7f26980b4d60 legacy=0x7f26981dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 >> v1:172.21.15.76:6789/0 conn(0x7f26980b1730 legacy=0x7f2698134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 1 monclient: found mon.a 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2698137a90 con 0x7f269814be90 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 10 monclient: _finish_auth 0 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f2694ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 10 monclient: _renew_subs 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2698137ce0 con 0x7f269814be90 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 10 monclient: _renew_subs 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:19.235 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2698145680 con 0x7f269814be90 2024-03-25T19:51:19.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.233+0000 7f269ea0e640 1 librados: init done 2024-03-25T19:51:19.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.234+0000 7f269ea0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.234+0000 7f269ea0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.234+0000 7f269ea0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:19.236 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.234+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26981403d0 con 0x7f269814be90 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.555+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2690003c20 con 0x7f269814be90 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.555+0000 7f2694ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.555+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2690003f30 con 0x7f269814be90 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.555+0000 7f2694ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.555+0000 7f2694ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.555+0000 7f26967fc640 4 set_mon_vals no callback set 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.555+0000 7f2694ff9640 10 monclient: dump: 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:19.556 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:19.557 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.556+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f2690019f00 con 0x7f269814be90 2024-03-25T19:51:19.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.557+0000 7f2697fff640 10 monclient: get_auth_request con 0x7f267806dd20 auth_method 0 2024-03-25T19:51:19.558 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.557+0000 7f2694ff9640 10 client.4713.objecter ms_handle_connect 0x7f267806dd20 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7f2690053920 con 0x7f269814be90 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 10 client.4713.objecter ms_dispatch 0x7f2698000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f269004f430 con 0x7f269814be90 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.564+0000 7f2694ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:19.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.818+0000 7f269ea0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.818+0000 7f269ea0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.818+0000 7f269ea0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:19.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.818+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26980627b0 con 0x7f269814be90 2024-03-25T19:51:19.879 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.879+0000 7f2694ff9640 1 -- 172.21.15.50:0/2607807550 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7f2690051360 con 0x7f269814be90 2024-03-25T19:51:19.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.879+0000 7f2694ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.879+0000 7f2694ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:19.880 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:19.880 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:19.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.881+0000 7f269ea0e640 10 librados: watch_flush enter 2024-03-25T19:51:19.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 10 librados: watch_flush exit 2024-03-25T19:51:19.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T19:51:19.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 10 client.4713.objecter successfully canceled tick 2024-03-25T19:51:19.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 >> v1:172.21.15.76:6800/2572556811 conn(0x7f267806dd20 legacy=0x7f26780701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 10 monclient: shutdown 2024-03-25T19:51:19.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:19.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 >> v1:172.21.15.50:6789/0 conn(0x7f269814be90 legacy=0x7f26981db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 shutdown_connections 2024-03-25T19:51:19.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 >> 172.21.15.50:0/2607807550 conn(0x7f26980617a0 msgr2=0x7f269813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:19.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 shutdown_connections 2024-03-25T19:51:19.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 1 -- 172.21.15.50:0/2607807550 wait complete. 2024-03-25T19:51:19.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:19.882+0000 7f269ea0e640 1 librados: shutdown 2024-03-25T19:51:19.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:26.343 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:46.849 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.346 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:46.852 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.378 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:10:46.884 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.420 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:46.891 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.427 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:46.898 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.431 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:46.936 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.436 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:46.941 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.447 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:10:46.918 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.471 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:10:46.977 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:26.701 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:26.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.816+0000 7f8979c25640 10 monclient: get_monmap_and_config 2024-03-25T19:51:26.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.816+0000 7f8979c25640 10 monclient: build_initial_monmap 2024-03-25T19:51:26.815 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.816+0000 7f8979c25640 10 monclient: monmap: 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:26.816888+0000 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:26.816888+0000 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:26.816 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:26.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.816+0000 7f8979c25640 1 Processor -- start 2024-03-25T19:51:26.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.816+0000 7f8979c25640 1 -- start start 2024-03-25T19:51:26.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.816+0000 7f8979c25640 10 monclient: init 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 10 monclient(hunting): picked mon.noname-a con 0x7f89740a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 10 monclient(hunting): picked mon.noname-c con 0x7f897413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 10 monclient(hunting): picked mon.noname-b con 0x7f89741430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8974142520 con 0x7f89740a98f0 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89740b28a0 con 0x7f897413f710 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89740b2a30 con 0x7f89741430c0 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:26.818 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.817+0000 7f8979c25640 10 monclient(hunting): authenticate will time out at 5462.081213s 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.826+0000 7f8972ffd640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f897413f710 0x7f89741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:45280/0 (socket says 172.21.15.50:45280) 2024-03-25T19:51:26.825 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.826+0000 7f8972ffd640 1 -- 172.21.15.50:0/983792160 learned_addr learned my addr 172.21.15.50:0/983792160 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:26.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.836+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445233092 0 0) 0x7f89740b28a0 con 0x7f897413f710 2024-03-25T19:51:26.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.836+0000 7f89727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.836+0000 7f89727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.836+0000 7f89727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.836+0000 7f89727fc640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:51:26.836 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.836+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895c003520 con 0x7f897413f710 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.844+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605966284 0 0) 0x7f8974142520 con 0x7f89740a98f0 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.844+0000 7f89727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.843 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.844+0000 7f89727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.844+0000 7f89727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.844+0000 7f89727fc640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:51:26.844 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.844+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89740b28a0 con 0x7f89740a98f0 2024-03-25T19:51:26.850 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.851+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3550766850 0 0) 0x7f89740b2a30 con 0x7f89741430c0 2024-03-25T19:51:26.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.851+0000 7f89727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.851+0000 7f89727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.851+0000 7f89727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.851+0000 7f89727fc640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:51:26.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.851+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8974142520 con 0x7f89741430c0 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63025520 0 0) 0x7f89740b28a0 con 0x7f89740a98f0 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 >> v1:172.21.15.50:6790/0 conn(0x7f89741430c0 legacy=0x7f897414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 >> v1:172.21.15.76:6789/0 conn(0x7f897413f710 legacy=0x7f89741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 1 monclient: found mon.noname-a 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:26.855 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89740b44c0 con 0x7f89740a98f0 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8964001670 con 0x7f89740a98f0 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f8979c25640 5 monclient: authenticate success, global_id 4716 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f8979c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f89727fc640 10 monclient: dump: 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.856 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.857 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.857 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:26.857 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:26.857 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:26.857 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:26.857 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:26.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.855+0000 7f8979c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:26.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89640017d0 con 0x7f89740a98f0 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f89727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f89727fc640 1 -- 172.21.15.50:0/983792160 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8964001be0 con 0x7f89740a98f0 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f8979c25640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f8979c25640 4 set_mon_vals no callback set 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f89727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f89727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f89727fc640 10 monclient: dump: 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:26.877 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:26.878 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:26.878 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:26.878 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f8979c25640 10 monclient: shutdown 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f8979c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.877+0000 7f8979c25640 1 -- 172.21.15.50:0/983792160 >> v1:172.21.15.50:6789/0 conn(0x7f89740a98f0 legacy=0x7f89740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.878+0000 7f8979c25640 1 -- 172.21.15.50:0/983792160 shutdown_connections 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.878+0000 7f8979c25640 1 -- 172.21.15.50:0/983792160 >> 172.21.15.50:0/983792160 conn(0x7f89740b7ea0 msgr2=0x7f89740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.878+0000 7f8979c25640 1 -- 172.21.15.50:0/983792160 shutdown_connections 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.878+0000 7f8979c25640 1 -- 172.21.15.50:0/983792160 wait complete. 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.878+0000 7f8979c25640 10 monclient: build_initial_monmap 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.878+0000 7f8979c25640 10 monclient: monmap: 2024-03-25T19:51:26.880 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 librados: starting msgr at 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 librados: starting objecter 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 Processor -- start 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 -- start start 2024-03-25T19:51:26.881 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 librados: setting wanted keys 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 librados: calling monclient init 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient: init 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient(hunting): picked mon.noname-a con 0x7f89740a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient(hunting): picked mon.noname-c con 0x7f897413f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient(hunting): picked mon.noname-b con 0x7f89741430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89741dad60 con 0x7f89740a98f0 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89740b1730 con 0x7f89741430c0 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f897414de50 con 0x7f897413f710 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:26.882 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.879+0000 7f8979c25640 10 monclient(hunting): authenticate will time out at 5462.143655s 2024-03-25T19:51:26.891 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.891+0000 7f89737fe640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f89740a98f0 0x7f89741db1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48194/0 (socket says 172.21.15.50:48194) 2024-03-25T19:51:26.891 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.891+0000 7f89737fe640 1 -- 172.21.15.50:0/2639901186 learned_addr learned my addr 172.21.15.50:0/2639901186 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:26.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.898+0000 7f895bfff640 10 client.?.objecter ms_handle_connect 0x7f89740a98f0 2024-03-25T19:51:26.897 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.898+0000 7f895bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.900+0000 7f895bfff640 10 client.?.objecter ms_handle_connect 0x7f89741430c0 2024-03-25T19:51:26.900 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.900+0000 7f895bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.906 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.906+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515222227 0 0) 0x7f89740b1730 con 0x7f89741430c0 2024-03-25T19:51:26.906 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.906+0000 7f895bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.906 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.906+0000 7f895bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.906 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.906+0000 7f895bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.906 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.906+0000 7f895bfff640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:51:26.906 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.906+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8960004eb0 con 0x7f89741430c0 2024-03-25T19:51:26.911 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.912+0000 7f895bfff640 10 client.?.objecter ms_handle_connect 0x7f897413f710 2024-03-25T19:51:26.912 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.912+0000 7f895bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.920+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696573308 0 0) 0x7f89741dad60 con 0x7f89740a98f0 2024-03-25T19:51:26.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.920+0000 7f895bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.920+0000 7f895bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.920+0000 7f895bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.920+0000 7f895bfff640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:51:26.920 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.920+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89740b1730 con 0x7f89740a98f0 2024-03-25T19:51:26.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.926+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734244988 0 0) 0x7f897414de50 con 0x7f897413f710 2024-03-25T19:51:26.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.926+0000 7f895bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.926+0000 7f895bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.925 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.926+0000 7f895bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.926+0000 7f895bfff640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:51:26.926 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.926+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89741dad60 con 0x7f897413f710 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.935+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481247801 0 0) 0x7f8960004eb0 con 0x7f89741430c0 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.935+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f897414de50 con 0x7f89741430c0 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.935+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f896c001bd0 con 0x7f89741430c0 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.935+0000 7f895bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.935+0000 7f895bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.935+0000 7f895bfff640 10 monclient(hunting): dump: 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:26.935 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:26.936 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.935+0000 7f895bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.949+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 291232012 0 0) 0x7f89740b1730 con 0x7f89740a98f0 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.949+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8960004eb0 con 0x7f89740a98f0 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.949+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f8964001670 con 0x7f89740a98f0 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.949+0000 7f895bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.949+0000 7f895bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.949+0000 7f895bfff640 10 monclient(hunting): dump: 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:26.949 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:26.950 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:26.950 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:26.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.957+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1291709328 0 0) 0x7f89741dad60 con 0x7f897413f710 2024-03-25T19:51:26.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.957+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89740b1730 con 0x7f897413f710 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.957+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f89680017b0 con 0x7f897413f710 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.957+0000 7f895bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.957+0000 7f895bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.957+0000 7f895bfff640 10 monclient(hunting): dump: 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:26.957 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:26.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3932134372 0 0) 0x7f897414de50 con 0x7f89741430c0 2024-03-25T19:51:26.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 >> v1:172.21.15.50:6790/0 conn(0x7f897413f710 legacy=0x7f89740b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 >> v1:172.21.15.50:6789/0 conn(0x7f89740a98f0 legacy=0x7f89741db1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 1 monclient: found mon.b 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f897414eb60 con 0x7f89741430c0 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 10 monclient: _finish_auth 0 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f895bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 10 monclient: _renew_subs 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89741da8f0 con 0x7f89741430c0 2024-03-25T19:51:26.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 10 monclient: _renew_subs 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f897414e2a0 con 0x7f89741430c0 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.961+0000 7f8979c25640 1 librados: init done 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.962+0000 7f8979c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.962+0000 7f8979c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.962+0000 7f8979c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:26.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:26.962+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89740b0a40 con 0x7f89741430c0 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.227+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f896c002060 con 0x7f89741430c0 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.227+0000 7f895bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.227+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f896c003b30 con 0x7f89741430c0 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.227+0000 7f895bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.227+0000 7f895bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.227+0000 7f895bfff640 10 monclient: dump: 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:27.227 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:27.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.227+0000 7f8971ffb640 4 set_mon_vals no callback set 2024-03-25T19:51:27.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.228+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f896c019b40 con 0x7f89741430c0 2024-03-25T19:51:27.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.229+0000 7f89737fe640 10 monclient: get_auth_request con 0x7f896006e0b0 auth_method 0 2024-03-25T19:51:27.229 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.229+0000 7f895bfff640 10 client.4696.objecter ms_handle_connect 0x7f896006e0b0 2024-03-25T19:51:27.233 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7f896c053840 con 0x7f89741430c0 2024-03-25T19:51:27.233 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 10 client.4696.objecter ms_dispatch 0x7f8974000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:27.233 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 3 client.4696.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:27.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 3 client.4696.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:27.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T19:51:27.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f896c04f180 con 0x7f89741430c0 2024-03-25T19:51:27.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:27.234 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.234+0000 7f895bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:27.468 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.468+0000 7f8979c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.469 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.468+0000 7f8979c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.469 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.469+0000 7f8979c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:27.469 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.469+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89741daf10 con 0x7f89741430c0 2024-03-25T19:51:27.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.614+0000 7f895bfff640 1 -- 172.21.15.50:0/2639901186 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7f89741daf10 con 0x7f89741430c0 2024-03-25T19:51:27.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.614+0000 7f895bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.614 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.614+0000 7f895bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:27.614 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:27.614 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:27.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 10 librados: watch_flush enter 2024-03-25T19:51:27.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 10 librados: watch_flush exit 2024-03-25T19:51:27.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T19:51:27.616 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 10 client.4696.objecter successfully canceled tick 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 >> v1:172.21.15.76:6800/2572556811 conn(0x7f896006e0b0 legacy=0x7f8960070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 10 monclient: shutdown 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 >> v1:172.21.15.76:6789/0 conn(0x7f89741430c0 legacy=0x7f89741e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 shutdown_connections 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 >> 172.21.15.50:0/2639901186 conn(0x7f89740b7ea0 msgr2=0x7f8974074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 shutdown_connections 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.616+0000 7f8979c25640 1 -- 172.21.15.50:0/2639901186 wait complete. 2024-03-25T19:51:27.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:27.617+0000 7f8979c25640 1 librados: shutdown 2024-03-25T19:51:27.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:34.434 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:34.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.553+0000 7f265b17a640 10 monclient: get_monmap_and_config 2024-03-25T19:51:34.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.553+0000 7f265b17a640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.553 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.553+0000 7f265b17a640 10 monclient: monmap: 2024-03-25T19:51:34.553 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:34.554728+0000 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:34.554728+0000 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:34.554 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:34.555 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.554+0000 7f265b17a640 1 Processor -- start 2024-03-25T19:51:34.555 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.554+0000 7f265b17a640 1 -- start start 2024-03-25T19:51:34.555 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.554+0000 7f265b17a640 10 monclient: init 2024-03-25T19:51:34.555 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 10 monclient(hunting): picked mon.noname-b con 0x7f26540a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 10 monclient(hunting): picked mon.noname-c con 0x7f265413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 10 monclient(hunting): picked mon.noname-a con 0x7f26541430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2654142520 con 0x7f26541430c0 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26540b28a0 con 0x7f265413f710 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26540b2a30 con 0x7f26540a98f0 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.556 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.555+0000 7f265b17a640 10 monclient(hunting): authenticate will time out at 5469.819591s 2024-03-25T19:51:34.560 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.560+0000 7f2659717640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f26541430c0 0x7f265414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:43782/0 (socket says 172.21.15.50:43782) 2024-03-25T19:51:34.560 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.560+0000 7f2659717640 1 -- 172.21.15.50:0/1562319110 learned_addr learned my addr 172.21.15.50:0/1562319110 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.569+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458395795 0 0) 0x7f2654142520 con 0x7f26541430c0 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.569+0000 7f26537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.569+0000 7f26537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.569+0000 7f26537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.569+0000 7f26537fe640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.569+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2644003520 con 0x7f26541430c0 2024-03-25T19:51:34.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.577+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188395083 0 0) 0x7f26540b2a30 con 0x7f26540a98f0 2024-03-25T19:51:34.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.577+0000 7f26537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.577+0000 7f26537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.577+0000 7f26537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.577+0000 7f26537fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:51:34.577 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.577+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2654142520 con 0x7f26540a98f0 2024-03-25T19:51:34.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.599+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790302601 0 0) 0x7f26540b28a0 con 0x7f265413f710 2024-03-25T19:51:34.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.599+0000 7f26537fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.599+0000 7f26537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.599+0000 7f26537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.599+0000 7f26537fe640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:51:34.599 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.599+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26540b2a30 con 0x7f265413f710 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 666958210 0 0) 0x7f2644003520 con 0x7f26541430c0 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 >> v1:172.21.15.50:6790/0 conn(0x7f26540a98f0 legacy=0x7f26540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 >> v1:172.21.15.76:6789/0 conn(0x7f265413f710 legacy=0x7f26541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 1 monclient: found mon.noname-a 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540b44c0 con 0x7f26541430c0 2024-03-25T19:51:34.600 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f264c001a10 con 0x7f26541430c0 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f26537fe640 10 monclient: dump: 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:34.601 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:34.602 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:34.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f265b17a640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:51:34.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.600+0000 7f265b17a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f264c001bf0 con 0x7f26541430c0 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f26537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f26537fe640 1 -- 172.21.15.50:0/1562319110 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f264c002000 con 0x7f26541430c0 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f265b17a640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f265b17a640 4 set_mon_vals no callback set 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f26537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f26537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.624+0000 7f26537fe640 10 monclient: dump: 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:34.624 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:34.625 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:34.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.625+0000 7f265b17a640 10 monclient: shutdown 2024-03-25T19:51:34.626 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.625+0000 7f265b17a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.625+0000 7f265b17a640 1 -- 172.21.15.50:0/1562319110 >> v1:172.21.15.50:6789/0 conn(0x7f26541430c0 legacy=0x7f265414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.625+0000 7f265b17a640 1 -- 172.21.15.50:0/1562319110 shutdown_connections 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.625+0000 7f265b17a640 1 -- 172.21.15.50:0/1562319110 >> 172.21.15.50:0/1562319110 conn(0x7f26540b7ea0 msgr2=0x7f26540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.625+0000 7f265b17a640 1 -- 172.21.15.50:0/1562319110 shutdown_connections 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.625+0000 7f265b17a640 1 -- 172.21.15.50:0/1562319110 wait complete. 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.626+0000 7f265b17a640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.626+0000 7f265b17a640 10 monclient: monmap: 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:34.627 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 1 librados: starting msgr at 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 1 librados: starting objecter 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 1 Processor -- start 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 1 -- start start 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 1 librados: setting wanted keys 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 1 librados: calling monclient init 2024-03-25T19:51:34.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 10 monclient: init 2024-03-25T19:51:34.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:34.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.627+0000 7f265b17a640 10 monclient(hunting): picked mon.noname-b con 0x7f26540a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:34.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.628+0000 7f265b17a640 10 monclient(hunting): picked mon.noname-a con 0x7f265413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:34.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.628+0000 7f265b17a640 10 monclient(hunting): picked mon.noname-c con 0x7f26541430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:34.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.628+0000 7f265b17a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2654139830 con 0x7f265413f710 2024-03-25T19:51:34.629 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.628+0000 7f265b17a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26541db7d0 con 0x7f26540a98f0 2024-03-25T19:51:34.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.628+0000 7f265b17a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26541db410 con 0x7f26541430c0 2024-03-25T19:51:34.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.628+0000 7f265b17a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.630 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.628+0000 7f265b17a640 10 monclient(hunting): authenticate will time out at 5469.891897s 2024-03-25T19:51:34.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.634+0000 7f2658f16640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f26540a98f0 0x7f26541dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:36170/0 (socket says 172.21.15.50:36170) 2024-03-25T19:51:34.634 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.634+0000 7f2658f16640 1 -- 172.21.15.50:0/2611751565 learned_addr learned my addr 172.21.15.50:0/2611751565 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:34.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.635+0000 7f26517fa640 10 client.?.objecter ms_handle_connect 0x7f26540a98f0 2024-03-25T19:51:34.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.635+0000 7f26517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.640+0000 7f26517fa640 10 client.?.objecter ms_handle_connect 0x7f265413f710 2024-03-25T19:51:34.640 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.640+0000 7f26517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.641+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276224462 0 0) 0x7f26541db7d0 con 0x7f26540a98f0 2024-03-25T19:51:34.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.641+0000 7f26517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.641+0000 7f26517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.641+0000 7f26517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.641+0000 7f26517fa640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:51:34.641 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.641+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2638004eb0 con 0x7f26540a98f0 2024-03-25T19:51:34.648 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 10 client.?.objecter ms_handle_connect 0x7f26541430c0 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949163606 0 0) 0x7f2654139830 con 0x7f265413f710 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:51:34.649 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.649+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26541db7d0 con 0x7f265413f710 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.653+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2681371763 0 0) 0x7f2638004eb0 con 0x7f26540a98f0 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.653+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2654139830 con 0x7f26540a98f0 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.653+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f2648001990 con 0x7f26540a98f0 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.653+0000 7f26517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.653+0000 7f26517fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.653+0000 7f26517fa640 10 monclient(hunting): dump: 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.653 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:34.654 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.653+0000 7f26517fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.657+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3101216200 0 0) 0x7f2654139830 con 0x7f26540a98f0 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.657+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 >> v1:172.21.15.50:6790/0 conn(0x7f26541430c0 legacy=0x7f26541d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 >> v1:172.21.15.50:6789/0 conn(0x7f265413f710 legacy=0x7f26540b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 1 monclient: found mon.b 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26541db5a0 con 0x7f26540a98f0 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.658 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f26517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 5 monclient: authenticate success, global_id 4702 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 10 monclient: _renew_subs 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26540b1480 con 0x7f26540a98f0 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 10 monclient: _renew_subs 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26540b0b10 con 0x7f26540a98f0 2024-03-25T19:51:34.659 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.658+0000 7f265b17a640 1 librados: init done 2024-03-25T19:51:34.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.659+0000 7f265b17a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.659+0000 7f265b17a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.659+0000 7f265b17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:34.660 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.659+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26541db960 con 0x7f26540a98f0 2024-03-25T19:51:34.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.948+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26480035f0 con 0x7f26540a98f0 2024-03-25T19:51:34.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.948+0000 7f26517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:34.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.948+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f26480039c0 con 0x7f26540a98f0 2024-03-25T19:51:34.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.948+0000 7f26517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.948+0000 7f26517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:34.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.948+0000 7f26517fa640 10 monclient: dump: 2024-03-25T19:51:34.948 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:34.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.948+0000 7f2652ffd640 4 set_mon_vals no callback set 2024-03-25T19:51:34.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.949+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f2648003be0 con 0x7f26540a98f0 2024-03-25T19:51:34.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.949+0000 7f2653fff640 10 monclient: get_auth_request con 0x7f263806dd10 auth_method 0 2024-03-25T19:51:34.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.950+0000 7f26517fa640 10 client.4702.objecter ms_handle_connect 0x7f263806dd10 2024-03-25T19:51:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.955+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7f26480547b0 con 0x7f26540a98f0 2024-03-25T19:51:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.955+0000 7f26517fa640 10 client.4702.objecter ms_dispatch 0x7f2654000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.955+0000 7f26517fa640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.955+0000 7f26517fa640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.955+0000 7f26517fa640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T19:51:34.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.956+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26480502c0 con 0x7f26540a98f0 2024-03-25T19:51:34.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.956+0000 7f26517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.956 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:34.956+0000 7f26517fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:35.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.192+0000 7f265b17a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.192+0000 7f265b17a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.192+0000 7f265b17a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:35.192 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.192+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26540627b0 con 0x7f26540a98f0 2024-03-25T19:51:35.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.276+0000 7f26517fa640 1 -- 172.21.15.50:0/2611751565 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7f26480521f0 con 0x7f26540a98f0 2024-03-25T19:51:35.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.276+0000 7f26517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:35.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.276+0000 7f26517fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:35.276 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:35.276 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 10 librados: watch_flush enter 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 10 librados: watch_flush exit 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 10 client.4702.objecter successfully canceled tick 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 >> v1:172.21.15.76:6800/2572556811 conn(0x7f263806dd10 legacy=0x7f26380701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 10 monclient: shutdown 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.278+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 >> v1:172.21.15.76:6789/0 conn(0x7f26540a98f0 legacy=0x7f26541dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.279+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 shutdown_connections 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.279+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 >> 172.21.15.50:0/2611751565 conn(0x7f26540b7ea0 msgr2=0x7f2654074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.279+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 shutdown_connections 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.279+0000 7f265b17a640 1 -- 172.21.15.50:0/2611751565 wait complete. 2024-03-25T19:51:35.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:35.279+0000 7f265b17a640 1 librados: shutdown 2024-03-25T19:51:35.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:42.098 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.213+0000 7fdf4544a640 10 monclient: get_monmap_and_config 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.213+0000 7fdf4544a640 10 monclient: build_initial_monmap 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.213+0000 7fdf4544a640 10 monclient: monmap: 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:42.214407+0000 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:42.214407+0000 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:42.213 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:42.214 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.214+0000 7fdf4544a640 1 Processor -- start 2024-03-25T19:51:42.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.214+0000 7fdf4544a640 1 -- start start 2024-03-25T19:51:42.214 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.214+0000 7fdf4544a640 10 monclient: init 2024-03-25T19:51:42.215 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:42.215 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 10 monclient(hunting): picked mon.noname-b con 0x7fdf4006b940 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 10 monclient(hunting): picked mon.noname-a con 0x7fdf400aed50 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 10 monclient(hunting): picked mon.noname-c con 0x7fdf4013c9c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4006f600 con 0x7fdf400aed50 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf40071210 con 0x7fdf4013c9c0 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf400714f0 con 0x7fdf4006b940 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:42.216 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.215+0000 7fdf4544a640 10 monclient(hunting): authenticate will time out at 5477.479590s 2024-03-25T19:51:42.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.221+0000 7fdf37fff640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fdf400aed50 0x7fdf40071bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:51860/0 (socket says 172.21.15.50:51860) 2024-03-25T19:51:42.221 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.221+0000 7fdf37fff640 1 -- 172.21.15.50:0/2350545052 learned_addr learned my addr 172.21.15.50:0/2350545052 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:42.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.242+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679245297 0 0) 0x7fdf40071210 con 0x7fdf4013c9c0 2024-03-25T19:51:42.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.242+0000 7fdf3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.242+0000 7fdf3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.242+0000 7fdf3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.242+0000 7fdf3e7fc640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:51:42.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.242+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf2c003520 con 0x7fdf4013c9c0 2024-03-25T19:51:42.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.245+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371608157 0 0) 0x7fdf4006f600 con 0x7fdf400aed50 2024-03-25T19:51:42.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.245+0000 7fdf3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.245+0000 7fdf3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.245+0000 7fdf3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.245+0000 7fdf3e7fc640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:51:42.245 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.245+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf40071210 con 0x7fdf400aed50 2024-03-25T19:51:42.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.253+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037935530 0 0) 0x7fdf400714f0 con 0x7fdf4006b940 2024-03-25T19:51:42.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.253+0000 7fdf3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.253+0000 7fdf3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.253+0000 7fdf3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.253+0000 7fdf3e7fc640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:51:42.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.253+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4006f600 con 0x7fdf4006b940 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3744980049 0 0) 0x7fdf2c003520 con 0x7fdf4013c9c0 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 >> v1:172.21.15.50:6790/0 conn(0x7fdf4006b940 legacy=0x7fdf4013c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 >> v1:172.21.15.50:6789/0 conn(0x7fdf400aed50 legacy=0x7fdf40071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf400717d0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.274 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fdf38001a10 con 0x7fdf4013c9c0 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf3e7fc640 10 monclient: dump: 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:42.275 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:42.276 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf4544a640 5 monclient: authenticate success, global_id 4705 2024-03-25T19:51:42.276 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.274+0000 7fdf4544a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf38001b70 con 0x7fdf4013c9c0 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf3e7fc640 1 -- 172.21.15.50:0/2350545052 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fdf38001f80 con 0x7fdf4013c9c0 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf3e7fc640 10 monclient: dump: 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:42.307 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf4544a640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:42.308 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.307+0000 7fdf4544a640 4 set_mon_vals no callback set 2024-03-25T19:51:42.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.308+0000 7fdf4544a640 10 monclient: shutdown 2024-03-25T19:51:42.309 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.308+0000 7fdf4544a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.308+0000 7fdf4544a640 1 -- 172.21.15.50:0/2350545052 >> v1:172.21.15.76:6789/0 conn(0x7fdf4013c9c0 legacy=0x7fdf40072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.308+0000 7fdf4544a640 1 -- 172.21.15.50:0/2350545052 shutdown_connections 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.308+0000 7fdf4544a640 1 -- 172.21.15.50:0/2350545052 >> 172.21.15.50:0/2350545052 conn(0x7fdf4006ab80 msgr2=0x7fdf4006af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.308+0000 7fdf4544a640 1 -- 172.21.15.50:0/2350545052 shutdown_connections 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.308+0000 7fdf4544a640 1 -- 172.21.15.50:0/2350545052 wait complete. 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.309+0000 7fdf4544a640 10 monclient: build_initial_monmap 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.309+0000 7fdf4544a640 10 monclient: monmap: 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:42.310 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 1 librados: starting msgr at 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 1 librados: starting objecter 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 1 Processor -- start 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 1 -- start start 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 1 librados: setting wanted keys 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 1 librados: calling monclient init 2024-03-25T19:51:42.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 10 monclient: init 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.310+0000 7fdf4544a640 10 monclient(hunting): picked mon.noname-c con 0x7fdf4006b940 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.311+0000 7fdf4544a640 10 monclient(hunting): picked mon.noname-b con 0x7fdf400aed50 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.311+0000 7fdf4544a640 10 monclient(hunting): picked mon.noname-a con 0x7fdf4013c9c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.311+0000 7fdf4544a640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf40142010 con 0x7fdf4013c9c0 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.311+0000 7fdf4544a640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf401d6ae0 con 0x7fdf400aed50 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.311+0000 7fdf4544a640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf401d6700 con 0x7fdf4006b940 2024-03-25T19:51:42.312 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.311+0000 7fdf4544a640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:42.313 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.311+0000 7fdf4544a640 10 monclient(hunting): authenticate will time out at 5477.574828s 2024-03-25T19:51:42.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.321+0000 7fdf3effd640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7fdf4006b940 0x7fdf401d7000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:53806/0 (socket says 172.21.15.50:53806) 2024-03-25T19:51:42.321 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.322+0000 7fdf3effd640 1 -- 172.21.15.50:0/2230682733 learned_addr learned my addr 172.21.15.50:0/2230682733 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:42.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.323+0000 7fdf377fe640 10 client.?.objecter ms_handle_connect 0x7fdf4006b940 2024-03-25T19:51:42.323 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.323+0000 7fdf377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.326+0000 7fdf377fe640 10 client.?.objecter ms_handle_connect 0x7fdf400aed50 2024-03-25T19:51:42.325 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.326+0000 7fdf377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.335+0000 7fdf377fe640 10 client.?.objecter ms_handle_connect 0x7fdf4013c9c0 2024-03-25T19:51:42.335 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.335+0000 7fdf377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.337+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013244073 0 0) 0x7fdf401d6700 con 0x7fdf4006b940 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.337+0000 7fdf377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.337+0000 7fdf377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.337+0000 7fdf377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.337+0000 7fdf377fe640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:51:42.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.337+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf20004eb0 con 0x7fdf4006b940 2024-03-25T19:51:42.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.348+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160678829 0 0) 0x7fdf40142010 con 0x7fdf4013c9c0 2024-03-25T19:51:42.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.348+0000 7fdf377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.348+0000 7fdf377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.348+0000 7fdf377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.348+0000 7fdf377fe640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:51:42.348 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.348+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf401d6700 con 0x7fdf4013c9c0 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.349+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351876088 0 0) 0x7fdf401d6ae0 con 0x7fdf400aed50 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.349+0000 7fdf377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.349+0000 7fdf377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.349+0000 7fdf377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.349+0000 7fdf377fe640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:51:42.349 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.349+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf40142010 con 0x7fdf400aed50 2024-03-25T19:51:42.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2758632736 0 0) 0x7fdf20004eb0 con 0x7fdf4006b940 2024-03-25T19:51:42.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf401d6ae0 con 0x7fdf4006b940 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fdf30001a10 con 0x7fdf4006b940 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 10 monclient(hunting): dump: 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:42.362 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4037708133 0 0) 0x7fdf401d6700 con 0x7fdf4013c9c0 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf20004eb0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fdf38001e80 con 0x7fdf4013c9c0 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.361+0000 7fdf377fe640 10 monclient(hunting): dump: 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:42.363 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:42.364 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.376+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2540979156 0 0) 0x7fdf40142010 con 0x7fdf400aed50 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.376+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf401d6700 con 0x7fdf400aed50 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.376+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fdf28001930 con 0x7fdf400aed50 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.376+0000 7fdf377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.376+0000 7fdf377fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.376+0000 7fdf377fe640 10 monclient(hunting): dump: 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.376 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:42.377 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2413861381 0 0) 0x7fdf20004eb0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 >> v1:172.21.15.50:6790/0 conn(0x7fdf4006b940 legacy=0x7fdf401d7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 >> v1:172.21.15.76:6789/0 conn(0x7fdf400aed50 legacy=0x7fdf401d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 1 monclient: found mon.a 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:42.385 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf401d6890 con 0x7fdf4013c9c0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 5 monclient: authenticate success, global_id 4731 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 10 monclient: _renew_subs 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf401d0b40 con 0x7fdf4013c9c0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 10 monclient: _renew_subs 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:42.386 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf401d0df0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.385+0000 7fdf4544a640 1 librados: init done 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.386+0000 7fdf4544a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.386+0000 7fdf4544a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.386+0000 7fdf4544a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:42.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.386+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf40070a20 con 0x7fdf4013c9c0 2024-03-25T19:51:42.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.691+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf38001fe0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.691+0000 7fdf377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:42.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.691+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fdf38003dc0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.691+0000 7fdf377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.691+0000 7fdf377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.691+0000 7fdf377fe640 10 monclient: dump: 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:42.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.691+0000 7fdf3dffb640 4 set_mon_vals no callback set 2024-03-25T19:51:42.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.692+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fdf38019d90 con 0x7fdf4013c9c0 2024-03-25T19:51:42.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.693+0000 7fdf3effd640 10 monclient: get_auth_request con 0x7fdf2006e110 auth_method 0 2024-03-25T19:51:42.693 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.693+0000 7fdf377fe640 10 client.4731.objecter ms_handle_connect 0x7fdf2006e110 2024-03-25T19:51:42.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.698+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7fdf380538e0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.698+0000 7fdf377fe640 10 client.4731.objecter ms_dispatch 0x7fdf40000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:42.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.698+0000 7fdf377fe640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:42.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.698+0000 7fdf377fe640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:42.698 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.698+0000 7fdf377fe640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T19:51:42.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.699+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf3804f3f0 con 0x7fdf4013c9c0 2024-03-25T19:51:42.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.699+0000 7fdf377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.699 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.699+0000 7fdf377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:42.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.940+0000 7fdf4544a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.940+0000 7fdf4544a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.940 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.940+0000 7fdf4544a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:42.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.940+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf401d6c70 con 0x7fdf4013c9c0 2024-03-25T19:51:42.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.984+0000 7fdf377fe640 1 -- 172.21.15.50:0/2230682733 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7fdf38051320 con 0x7fdf4013c9c0 2024-03-25T19:51:42.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.984+0000 7fdf377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.984+0000 7fdf377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:42.984 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:42.984 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 10 librados: watch_flush enter 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 10 librados: watch_flush exit 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 10 client.4731.objecter successfully canceled tick 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 >> v1:172.21.15.76:6800/2572556811 conn(0x7fdf2006e110 legacy=0x7fdf200705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 10 monclient: shutdown 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 >> v1:172.21.15.50:6789/0 conn(0x7fdf4013c9c0 legacy=0x7fdf40142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.987 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 shutdown_connections 2024-03-25T19:51:42.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 >> 172.21.15.50:0/2230682733 conn(0x7fdf4006ab80 msgr2=0x7fdf4013c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:42.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 shutdown_connections 2024-03-25T19:51:42.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.986+0000 7fdf4544a640 1 -- 172.21.15.50:0/2230682733 wait complete. 2024-03-25T19:51:42.988 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:42.987+0000 7fdf4544a640 1 librados: shutdown 2024-03-25T19:51:42.996 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:46.400 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:06.906 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.405 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:06.911 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.459 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:06.965 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.462 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:06.933 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.475 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:06.946 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.493 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:06.999 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.501 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:07.007 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.504 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:06.975 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.530 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:07.035 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:49.802 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:49.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.919+0000 7fed54f15640 10 monclient: get_monmap_and_config 2024-03-25T19:51:49.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.919+0000 7fed54f15640 10 monclient: build_initial_monmap 2024-03-25T19:51:49.919 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.919+0000 7fed54f15640 10 monclient: monmap: 2024-03-25T19:51:49.919 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:49.920730+0000 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:49.920730+0000 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:49.920 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.920+0000 7fed54f15640 1 Processor -- start 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.920+0000 7fed54f15640 1 -- start start 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.920+0000 7fed54f15640 10 monclient: init 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 10 monclient(hunting): picked mon.noname-b con 0x7fed500b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 10 monclient(hunting): picked mon.noname-a con 0x7fed500b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 10 monclient(hunting): picked mon.noname-c con 0x7fed5014be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed500af940 con 0x7fed500b1730 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed500b0a70 con 0x7fed5014be90 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed500b0d50 con 0x7fed500b4d60 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:49.922 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.921+0000 7fed54f15640 10 monclient(hunting): authenticate will time out at 5485.185113s 2024-03-25T19:51:49.923 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.923+0000 7fed4ed9d640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fed5014be90 0x7fed5014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:44580/0 (socket says 172.21.15.50:44580) 2024-03-25T19:51:49.923 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.923+0000 7fed4ed9d640 1 -- 172.21.15.50:0/3180201302 learned_addr learned my addr 172.21.15.50:0/3180201302 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:49.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.955+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039941440 0 0) 0x7fed500af940 con 0x7fed500b1730 2024-03-25T19:51:49.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.955+0000 7fed4d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.955+0000 7fed4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.955+0000 7fed4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.955+0000 7fed4d59a640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:51:49.955 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:49.955+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed38003520 con 0x7fed500b1730 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3344271063 0 0) 0x7fed38003520 con 0x7fed500b1730 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 >> v1:172.21.15.50:6790/0 conn(0x7fed500b4d60 legacy=0x7fed5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 >> v1:172.21.15.76:6789/0 conn(0x7fed5014be90 legacy=0x7fed5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 1 monclient: found mon.noname-a 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed500b1030 con 0x7fed500b1730 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 10 monclient: _finish_auth 0 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:51:50.021 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed44001690 con 0x7fed500b1730 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed4d59a640 10 monclient: dump: 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:50.022 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:50.023 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:50.023 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:50.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed54f15640 5 monclient: authenticate success, global_id 4734 2024-03-25T19:51:50.023 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.020+0000 7fed54f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed440017f0 con 0x7fed500b1730 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed4d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed4d59a640 1 -- 172.21.15.50:0/3180201302 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed44001c00 con 0x7fed500b1730 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed54f15640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed54f15640 4 set_mon_vals no callback set 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed4d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.038+0000 7fed4d59a640 10 monclient: dump: 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.038 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.039 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.039 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:50.039 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:50.039 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:50.039 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:50.039 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:50.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.039+0000 7fed54f15640 10 monclient: shutdown 2024-03-25T19:51:50.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.039+0000 7fed54f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:50.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.039+0000 7fed54f15640 1 -- 172.21.15.50:0/3180201302 >> v1:172.21.15.50:6789/0 conn(0x7fed500b1730 legacy=0x7fed500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.039+0000 7fed54f15640 1 -- 172.21.15.50:0/3180201302 shutdown_connections 2024-03-25T19:51:50.040 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.039+0000 7fed54f15640 1 -- 172.21.15.50:0/3180201302 >> 172.21.15.50:0/3180201302 conn(0x7fed500617a0 msgr2=0x7fed50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.039+0000 7fed54f15640 1 -- 172.21.15.50:0/3180201302 shutdown_connections 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.039+0000 7fed54f15640 1 -- 172.21.15.50:0/3180201302 wait complete. 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 10 monclient: build_initial_monmap 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 10 monclient: monmap: 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:50.041 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 1 librados: starting msgr at 2024-03-25T19:51:50.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 1 librados: starting objecter 2024-03-25T19:51:50.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 1 Processor -- start 2024-03-25T19:51:50.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 1 -- start start 2024-03-25T19:51:50.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 1 librados: setting wanted keys 2024-03-25T19:51:50.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 1 librados: calling monclient init 2024-03-25T19:51:50.042 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.040+0000 7fed54f15640 10 monclient: init 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 10 monclient(hunting): picked mon.noname-b con 0x7fed500b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 10 monclient(hunting): picked mon.noname-a con 0x7fed500b4d60 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 10 monclient(hunting): picked mon.noname-c con 0x7fed5014be90 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed50136e40 con 0x7fed500b4d60 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed501307c0 con 0x7fed500b1730 2024-03-25T19:51:50.043 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed50138610 con 0x7fed5014be90 2024-03-25T19:51:50.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:50.044 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.041+0000 7fed54f15640 10 monclient(hunting): authenticate will time out at 5485.305328s 2024-03-25T19:51:50.053 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.054+0000 7fed4dd9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7fed500b4d60 0x7fed5012f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:53482/0 (socket says 172.21.15.50:53482) 2024-03-25T19:51:50.054 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.054+0000 7fed4dd9b640 1 -- 172.21.15.50:0/2525351011 learned_addr learned my addr 172.21.15.50:0/2525351011 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:50.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.056+0000 7fed3effd640 10 client.?.objecter ms_handle_connect 0x7fed500b4d60 2024-03-25T19:51:50.057 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.056+0000 7fed3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.059+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877162484 0 0) 0x7fed50136e40 con 0x7fed500b4d60 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.059+0000 7fed3effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.059+0000 7fed3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.059+0000 7fed3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.059+0000 7fed3effd640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:51:50.059 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.059+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed30004eb0 con 0x7fed500b4d60 2024-03-25T19:51:50.079 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.080+0000 7fed3effd640 10 client.?.objecter ms_handle_connect 0x7fed500b1730 2024-03-25T19:51:50.080 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.080+0000 7fed3effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.085+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4091331873 0 0) 0x7fed30004eb0 con 0x7fed500b4d60 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.085+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed50136e40 con 0x7fed500b4d60 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.085+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed44001c00 con 0x7fed500b4d60 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.085+0000 7fed3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.085+0000 7fed3effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.085+0000 7fed3effd640 10 monclient(hunting): dump: 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.085 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.086 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.086 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:50.086 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:50.086 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:50.086 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:50.086 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:50.086 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.085+0000 7fed3effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:50.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.095+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997358606 0 0) 0x7fed501307c0 con 0x7fed500b1730 2024-03-25T19:51:50.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.095+0000 7fed3effd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:50.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.095+0000 7fed3effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:50.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.095+0000 7fed3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:50.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.095+0000 7fed3effd640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:51:50.095 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.095+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed30004eb0 con 0x7fed500b1730 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3191928058 0 0) 0x7fed50136e40 con 0x7fed500b4d60 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 >> v1:172.21.15.50:6790/0 conn(0x7fed5014be90 legacy=0x7fed501300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 >> v1:172.21.15.76:6789/0 conn(0x7fed500b1730 legacy=0x7fed501409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 1 monclient: found mon.a 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed501388f0 con 0x7fed500b4d60 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 10 monclient: _finish_auth 0 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed54f15640 5 monclient: authenticate success, global_id 4737 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed54f15640 10 monclient: _renew_subs 2024-03-25T19:51:50.097 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed54f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed501410e0 con 0x7fed500b4d60 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.096+0000 7fed54f15640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.097+0000 7fed54f15640 10 monclient: _renew_subs 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.097+0000 7fed54f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.097+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed501381a0 con 0x7fed500b4d60 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.097+0000 7fed54f15640 1 librados: init done 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.098+0000 7fed54f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:50.098 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.098+0000 7fed54f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:50.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.098+0000 7fed54f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:50.099 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.098+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed501401f0 con 0x7fed500b4d60 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.433+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed440038a0 con 0x7fed500b4d60 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.433+0000 7fed3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.433+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fed44003cb0 con 0x7fed500b4d60 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.433+0000 7fed3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.433+0000 7fed3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.433+0000 7fed3effd640 10 monclient: dump: 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:50.434 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:50.435 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:50.435 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:50.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.433+0000 7fed4cd99640 4 set_mon_vals no callback set 2024-03-25T19:51:50.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.434+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fed44003ed0 con 0x7fed500b4d60 2024-03-25T19:51:50.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.435+0000 7fed4e59c640 10 monclient: get_auth_request con 0x7fed3006dd30 auth_method 0 2024-03-25T19:51:50.435 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.435+0000 7fed3effd640 10 client.4737.objecter ms_handle_connect 0x7fed3006dd30 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7fed440549d0 con 0x7fed500b4d60 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 10 client.4737.objecter ms_dispatch 0x7fed50000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed440504e0 con 0x7fed500b4d60 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:50.440 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.440+0000 7fed3effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:50.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.691+0000 7fed54f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.691+0000 7fed54f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.691+0000 7fed54f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:51:50.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.691+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed500627b0 con 0x7fed500b4d60 2024-03-25T19:51:50.731 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.732+0000 7fed3effd640 1 -- 172.21.15.50:0/2525351011 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7fed44052410 con 0x7fed500b4d60 2024-03-25T19:51:50.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.732+0000 7fed3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.732 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.732+0000 7fed3effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:50.732 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:50.732 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:50.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 10 librados: watch_flush enter 2024-03-25T19:51:50.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 10 librados: watch_flush exit 2024-03-25T19:51:50.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T19:51:50.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 10 client.4737.objecter successfully canceled tick 2024-03-25T19:51:50.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 >> v1:172.21.15.76:6800/2572556811 conn(0x7fed3006dd30 legacy=0x7fed300701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 10 monclient: shutdown 2024-03-25T19:51:50.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:50.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 >> v1:172.21.15.50:6789/0 conn(0x7fed500b4d60 legacy=0x7fed5012f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 shutdown_connections 2024-03-25T19:51:50.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 >> 172.21.15.50:0/2525351011 conn(0x7fed500617a0 msgr2=0x7fed500b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:50.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 shutdown_connections 2024-03-25T19:51:50.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 1 -- 172.21.15.50:0/2525351011 wait complete. 2024-03-25T19:51:50.735 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:50.734+0000 7fed54f15640 1 librados: shutdown 2024-03-25T19:51:50.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:57.387 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:17.893 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.393 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:17.898 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.400 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:17.906 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.404 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:17.909 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.415 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:17.886 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.418 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:17.889 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.419 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:17.925 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.426 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:17.932 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.437 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:17.908 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.552 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.661+0000 7fbec3f5d640 10 monclient: get_monmap_and_config 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.661+0000 7fbec3f5d640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.661+0000 7fbec3f5d640 10 monclient: monmap: 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:51:57.662210+0000 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:51:57.662210+0000 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:51:57.661 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:51:57.662 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 1 Processor -- start 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 1 -- start start 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 10 monclient: init 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fbebc0b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fbebc0b1730 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fbebc14be90 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:57.663 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc0af940 con 0x7fbebc0b1730 2024-03-25T19:51:57.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.662+0000 7fbec3f5d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc0b0a70 con 0x7fbebc14be90 2024-03-25T19:51:57.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.663+0000 7fbec3f5d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc0b0d50 con 0x7fbebc0b4d60 2024-03-25T19:51:57.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.663+0000 7fbec3f5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.664 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.663+0000 7fbec3f5d640 10 monclient(hunting): authenticate will time out at 5492.926794s 2024-03-25T19:51:57.675 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.676+0000 7fbec24fa640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbebc14be90 0x7fbebc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:44286/0 (socket says 172.21.15.50:44286) 2024-03-25T19:51:57.676 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.676+0000 7fbec24fa640 1 -- 172.21.15.50:0/3253728652 learned_addr learned my addr 172.21.15.50:0/3253728652 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:57.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.702+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959323966 0 0) 0x7fbebc0b0d50 con 0x7fbebc0b4d60 2024-03-25T19:51:57.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.702+0000 7fbec0cf7640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.702+0000 7fbec0cf7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.702+0000 7fbec0cf7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.702+0000 7fbec0cf7640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:51:57.702 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.702+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea8003520 con 0x7fbebc0b4d60 2024-03-25T19:51:57.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.707+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008002857 0 0) 0x7fbebc0b0a70 con 0x7fbebc14be90 2024-03-25T19:51:57.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.707+0000 7fbec0cf7640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.707+0000 7fbec0cf7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.707+0000 7fbec0cf7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.707+0000 7fbec0cf7640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:51:57.707 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.707+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbebc0b0d50 con 0x7fbebc14be90 2024-03-25T19:51:57.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.712+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209234084 0 0) 0x7fbebc0af940 con 0x7fbebc0b1730 2024-03-25T19:51:57.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.712+0000 7fbec0cf7640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.712+0000 7fbec0cf7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.711 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.712+0000 7fbec0cf7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.712 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.712+0000 7fbec0cf7640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:51:57.712 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.712+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbebc0b0a70 con 0x7fbebc0b1730 2024-03-25T19:51:57.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 119309118 0 0) 0x7fbea8003520 con 0x7fbebc0b4d60 2024-03-25T19:51:57.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 >> v1:172.21.15.76:6789/0 conn(0x7fbebc14be90 legacy=0x7fbebc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 >> v1:172.21.15.50:6789/0 conn(0x7fbebc0b1730 legacy=0x7fbebc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 1 monclient: found mon.noname-b 2024-03-25T19:51:57.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.716 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc0b1030 con 0x7fbebc0b4d60 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbeb8001690 con 0x7fbebc0b4d60 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec3f5d640 5 monclient: authenticate success, global_id 4700 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec3f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec0cf7640 10 monclient: dump: 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:57.717 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:57.718 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.718 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:57.718 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:57.718 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:57.718 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:57.718 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:57.718 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.716+0000 7fbec3f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbeb80017f0 con 0x7fbebc0b4d60 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec0cf7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec0cf7640 1 -- 172.21.15.50:0/3253728652 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbeb8001c00 con 0x7fbebc0b4d60 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec3f5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec3f5d640 4 set_mon_vals no callback set 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec0cf7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec0cf7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:57.734 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.734+0000 7fbec0cf7640 10 monclient: dump: 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:57.735 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:57.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.735+0000 7fbec3f5d640 10 monclient: shutdown 2024-03-25T19:51:57.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.735+0000 7fbec3f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:57.737 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.735+0000 7fbec3f5d640 1 -- 172.21.15.50:0/3253728652 >> v1:172.21.15.50:6790/0 conn(0x7fbebc0b4d60 legacy=0x7fbebc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.735+0000 7fbec3f5d640 1 -- 172.21.15.50:0/3253728652 shutdown_connections 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.735+0000 7fbec3f5d640 1 -- 172.21.15.50:0/3253728652 >> 172.21.15.50:0/3253728652 conn(0x7fbebc0617a0 msgr2=0x7fbebc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.735+0000 7fbec3f5d640 1 -- 172.21.15.50:0/3253728652 shutdown_connections 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.735+0000 7fbec3f5d640 1 -- 172.21.15.50:0/3253728652 wait complete. 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 10 monclient: monmap: 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:51:57.738 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 1 librados: starting msgr at 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 1 librados: starting objecter 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 1 Processor -- start 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 1 -- start start 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 1 librados: setting wanted keys 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 1 librados: calling monclient init 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.736+0000 7fbec3f5d640 10 monclient: init 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fbebc0b1730 addr v1:172.21.15.76:6789/0 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fbebc0b4d60 addr v1:172.21.15.50:6790/0 2024-03-25T19:51:57.739 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fbebc14be90 addr v1:172.21.15.50:6789/0 2024-03-25T19:51:57.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc140f30 con 0x7fbebc14be90 2024-03-25T19:51:57.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc140a10 con 0x7fbebc0b1730 2024-03-25T19:51:57.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbebc135550 con 0x7fbebc0b4d60 2024-03-25T19:51:57.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.740 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.737+0000 7fbec3f5d640 10 monclient(hunting): authenticate will time out at 5493.001200s 2024-03-25T19:51:57.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.754+0000 7fbec1cf9640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbebc0b1730 0x7fbebc134e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:44310/0 (socket says 172.21.15.50:44310) 2024-03-25T19:51:57.754 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.754+0000 7fbec1cf9640 1 -- 172.21.15.50:0/1042583265 learned_addr learned my addr 172.21.15.50:0/1042583265 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:51:57.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.758+0000 7fbeb27fc640 10 client.?.objecter ms_handle_connect 0x7fbebc0b1730 2024-03-25T19:51:57.757 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.758+0000 7fbeb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.765+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323534386 0 0) 0x7fbebc140a10 con 0x7fbebc0b1730 2024-03-25T19:51:57.764 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.765+0000 7fbeb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.765+0000 7fbeb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.765+0000 7fbeb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.765+0000 7fbeb27fc640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:51:57.765 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.765+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0004eb0 con 0x7fbebc0b1730 2024-03-25T19:51:57.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.772+0000 7fbeb27fc640 10 client.?.objecter ms_handle_connect 0x7fbebc14be90 2024-03-25T19:51:57.771 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.772+0000 7fbeb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.778+0000 7fbeb27fc640 10 client.?.objecter ms_handle_connect 0x7fbebc0b4d60 2024-03-25T19:51:57.777 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.778+0000 7fbeb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.781+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3423792854 0 0) 0x7fbea0004eb0 con 0x7fbebc0b1730 2024-03-25T19:51:57.781 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.782+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbebc140a10 con 0x7fbebc0b1730 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.782+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbeb8001bc0 con 0x7fbebc0b1730 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.782+0000 7fbeb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.782+0000 7fbeb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.782+0000 7fbeb27fc640 10 monclient(hunting): dump: 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:57.782 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.782+0000 7fbeb27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.783+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242568635 0 0) 0x7fbebc140f30 con 0x7fbebc14be90 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.783+0000 7fbeb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.783+0000 7fbeb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.783 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.783+0000 7fbeb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.783+0000 7fbeb27fc640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:51:57.784 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.783+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0004eb0 con 0x7fbebc14be90 2024-03-25T19:51:57.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.785+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838247062 0 0) 0x7fbebc135550 con 0x7fbebc0b4d60 2024-03-25T19:51:57.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.785+0000 7fbeb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.785+0000 7fbeb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.785+0000 7fbeb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.785+0000 7fbeb27fc640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:51:57.785 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.785+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbebc140f30 con 0x7fbebc0b4d60 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3514465898 0 0) 0x7fbebc140a10 con 0x7fbebc0b1730 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 >> v1:172.21.15.50:6790/0 conn(0x7fbebc0b4d60 legacy=0x7fbebc1dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 >> v1:172.21.15.50:6789/0 conn(0x7fbebc14be90 legacy=0x7fbebc1e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 1 monclient: found mon.b 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc1356e0 con 0x7fbebc0b1730 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.799 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 10 monclient: _renew_subs 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbebc145160 con 0x7fbebc0b1730 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 10 monclient: _renew_subs 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbebc140560 con 0x7fbebc0b1730 2024-03-25T19:51:57.800 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.799+0000 7fbec3f5d640 1 librados: init done 2024-03-25T19:51:57.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.800+0000 7fbec3f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.800+0000 7fbec3f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.800+0000 7fbec3f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:57.801 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:57.800+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbebc145500 con 0x7fbebc0b1730 2024-03-25T19:51:58.072 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbeb80011f0 con 0x7fbebc0b1730 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7fbeb8003b00 con 0x7fbebc0b1730 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb27fc640 10 monclient: dump: 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:51:58.073 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:51:58.074 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:51:58.074 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:51:58.074 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:51:58.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb37fe640 4 set_mon_vals no callback set 2024-03-25T19:51:58.074 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.073+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7fbeb8003d20 con 0x7fbebc0b1730 2024-03-25T19:51:58.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.074+0000 7fbec14f8640 10 monclient: get_auth_request con 0x7fbea006de10 auth_method 0 2024-03-25T19:51:58.075 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.074+0000 7fbeb27fc640 10 client.4720.objecter ms_handle_connect 0x7fbea006de10 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7fbeb8054980 con 0x7fbebc0b1730 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 10 client.4720.objecter ms_dispatch 0x7fbebc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 3 client.4720.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 3 client.4720.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 20 client.4720.objecter dump_active .. 0 homeless 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbeb80502c0 con 0x7fbebc0b1730 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:58.078 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.078+0000 7fbeb27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:58.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.338+0000 7fbec3f5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.338+0000 7fbec3f5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.338+0000 7fbec3f5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:51:58.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.338+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbebc135870 con 0x7fbebc0b1730 2024-03-25T19:51:58.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.367+0000 7fbeb27fc640 1 -- 172.21.15.50:0/1042583265 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7fbeb80523c0 con 0x7fbebc0b1730 2024-03-25T19:51:58.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.367+0000 7fbeb27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:58.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.367+0000 7fbeb27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:58.367 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:51:58.367 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:58.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 10 librados: watch_flush enter 2024-03-25T19:51:58.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 10 librados: watch_flush exit 2024-03-25T19:51:58.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 20 client.4720.objecter shutdown clearing up homeless session... 2024-03-25T19:51:58.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 10 client.4720.objecter successfully canceled tick 2024-03-25T19:51:58.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 >> v1:172.21.15.76:6800/2572556811 conn(0x7fbea006de10 legacy=0x7fbea00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:58.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 10 monclient: shutdown 2024-03-25T19:51:58.369 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:58.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 >> v1:172.21.15.76:6789/0 conn(0x7fbebc0b1730 legacy=0x7fbebc134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:58.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 shutdown_connections 2024-03-25T19:51:58.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 >> 172.21.15.50:0/1042583265 conn(0x7fbebc0617a0 msgr2=0x7fbebc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:58.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 shutdown_connections 2024-03-25T19:51:58.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 1 -- 172.21.15.50:0/1042583265 wait complete. 2024-03-25T19:51:58.370 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:51:58.369+0000 7fbec3f5d640 1 librados: shutdown 2024-03-25T19:51:58.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:05.184 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:05.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.300+0000 7f7e30857640 10 monclient: get_monmap_and_config 2024-03-25T19:52:05.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.301+0000 7f7e30857640 10 monclient: build_initial_monmap 2024-03-25T19:52:05.301 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.301+0000 7f7e30857640 10 monclient: monmap: 2024-03-25T19:52:05.301 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:05.301 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:52:05.302844+0000 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:52:05.302844+0000 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:52:05.302 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:05.303 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.301+0000 7f7e30857640 1 Processor -- start 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.301+0000 7f7e30857640 1 -- start start 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.301+0000 7f7e30857640 10 monclient: init 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c142520 con 0x7f7e2c1430c0 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0b28a0 con 0x7f7e2c13f710 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0b2a30 con 0x7f7e2c0a98f0 2024-03-25T19:52:05.304 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:05.305 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.302+0000 7f7e30857640 10 monclient(hunting): authenticate will time out at 5500.567396s 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.310+0000 7f7e2a59c640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f7e2c0a98f0 0x7f7e2c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:36342/0 (socket says 172.21.15.50:36342) 2024-03-25T19:52:05.311 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.310+0000 7f7e2a59c640 1 -- 172.21.15.50:0/1937142463 learned_addr learned my addr 172.21.15.50:0/1937142463 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:05.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.331+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663879717 0 0) 0x7f7e2c0b28a0 con 0x7f7e2c13f710 2024-03-25T19:52:05.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.331+0000 7f7e29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.331+0000 7f7e29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.331+0000 7f7e29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.331+0000 7f7e29d9b640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:52:05.332 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.331+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18003520 con 0x7f7e2c13f710 2024-03-25T19:52:05.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.337+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490914278 0 0) 0x7f7e2c0b2a30 con 0x7f7e2c0a98f0 2024-03-25T19:52:05.337 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.337+0000 7f7e29d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.337+0000 7f7e29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.337+0000 7f7e29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.337+0000 7f7e29d9b640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:52:05.338 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.337+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c0b28a0 con 0x7f7e2c0a98f0 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2689493834 0 0) 0x7f7e18003520 con 0x7f7e2c13f710 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 >> v1:172.21.15.50:6790/0 conn(0x7f7e2c0a98f0 legacy=0x7f7e2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 >> v1:172.21.15.50:6789/0 conn(0x7f7e2c1430c0 legacy=0x7f7e2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 1 monclient: found mon.noname-c 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0b44c0 con 0x7f7e2c13f710 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 10 monclient: _finish_auth 0 2024-03-25T19:52:05.346 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7e14001690 con 0x7f7e2c13f710 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e30857640 5 monclient: authenticate success, global_id 4723 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e30857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e29d9b640 10 monclient: dump: 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:05.347 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:05.348 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:05.348 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:05.348 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:05.348 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.345+0000 7f7e30857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:05.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e140018b0 con 0x7f7e2c13f710 2024-03-25T19:52:05.361 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e29d9b640 1 -- 172.21.15.50:0/1937142463 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7e14001c40 con 0x7f7e2c13f710 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 4 set_mon_vals no callback set 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e29d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e29d9b640 10 monclient: dump: 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:05.362 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:05.363 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:05.363 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:05.364 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 10 monclient: shutdown 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 1 -- 172.21.15.50:0/1937142463 >> v1:172.21.15.76:6789/0 conn(0x7f7e2c13f710 legacy=0x7f7e2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 1 -- 172.21.15.50:0/1937142463 shutdown_connections 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 1 -- 172.21.15.50:0/1937142463 >> 172.21.15.50:0/1937142463 conn(0x7f7e2c0b7ea0 msgr2=0x7f7e2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 1 -- 172.21.15.50:0/1937142463 shutdown_connections 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.361+0000 7f7e30857640 1 -- 172.21.15.50:0/1937142463 wait complete. 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.362+0000 7f7e30857640 10 monclient: build_initial_monmap 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.362+0000 7f7e30857640 10 monclient: monmap: 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:52:05.365 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 1 librados: starting msgr at 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 1 librados: starting objecter 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 1 Processor -- start 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 1 -- start start 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 1 librados: setting wanted keys 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 1 librados: calling monclient init 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 10 monclient: init 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:05.366 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2c0a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.363+0000 7f7e30857640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2c13f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.364+0000 7f7e30857640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2c1430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.364+0000 7f7e30857640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c1d77a0 con 0x7f7e2c1430c0 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.364+0000 7f7e30857640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c138e40 con 0x7f7e2c13f710 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.364+0000 7f7e30857640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0aba30 con 0x7f7e2c0a98f0 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.364+0000 7f7e30857640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:05.367 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.364+0000 7f7e30857640 10 monclient(hunting): authenticate will time out at 5500.628735s 2024-03-25T19:52:05.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.370+0000 7f7e2ad9d640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7f7e2c1430c0 0x7f7e2c134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:48606/0 (socket says 172.21.15.50:48606) 2024-03-25T19:52:05.371 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.370+0000 7f7e2ad9d640 1 -- 172.21.15.50:0/150165049 learned_addr learned my addr 172.21.15.50:0/150165049 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:05.375 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.375+0000 7f7e22ffd640 10 client.?.objecter ms_handle_connect 0x7f7e2c13f710 2024-03-25T19:52:05.376 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.375+0000 7f7e22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.386+0000 7f7e22ffd640 10 client.?.objecter ms_handle_connect 0x7f7e2c1430c0 2024-03-25T19:52:05.387 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.386+0000 7f7e22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.389+0000 7f7e22ffd640 10 client.?.objecter ms_handle_connect 0x7f7e2c0a98f0 2024-03-25T19:52:05.390 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.389+0000 7f7e22ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:05.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.395+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095641461 0 0) 0x7f7e2c1d77a0 con 0x7f7e2c1430c0 2024-03-25T19:52:05.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.395+0000 7f7e22ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.395+0000 7f7e22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.395 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.395+0000 7f7e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.395+0000 7f7e22ffd640 10 monclient(hunting): my global_id is 4749 2024-03-25T19:52:05.396 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.395+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0c004eb0 con 0x7f7e2c1430c0 2024-03-25T19:52:05.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.396+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914195634 0 0) 0x7f7e2c138e40 con 0x7f7e2c13f710 2024-03-25T19:52:05.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.396+0000 7f7e22ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.396+0000 7f7e22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.396+0000 7f7e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.396+0000 7f7e22ffd640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:52:05.397 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.396+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c1d77a0 con 0x7f7e2c13f710 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.405+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036138000 0 0) 0x7f7e2c1d77a0 con 0x7f7e2c13f710 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.405+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e2c138e40 con 0x7f7e2c13f710 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.405+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7e140015d0 con 0x7f7e2c13f710 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.405+0000 7f7e22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.405+0000 7f7e22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.405+0000 7f7e22ffd640 10 monclient(hunting): dump: 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.406 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.407 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.407 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:05.407 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:05.407 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:05.407 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:05.407 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:05.407 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.405+0000 7f7e22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:05.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3560603560 0 0) 0x7f7e0c004eb0 con 0x7f7e2c1430c0 2024-03-25T19:52:05.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e2c1d77a0 con 0x7f7e2c1430c0 2024-03-25T19:52:05.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7e24001c30 con 0x7f7e2c1430c0 2024-03-25T19:52:05.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.412 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 10 monclient(hunting): dump: 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626392246 0 0) 0x7f7e2c0aba30 con 0x7f7e2c0a98f0 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:05.413 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:05.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:52:05.414 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.411+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0c004eb0 con 0x7f7e2c0a98f0 2024-03-25T19:52:05.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1797083317 0 0) 0x7f7e2c138e40 con 0x7f7e2c13f710 2024-03-25T19:52:05.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 >> v1:172.21.15.50:6790/0 conn(0x7f7e2c0a98f0 legacy=0x7f7e2c138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 >> v1:172.21.15.50:6789/0 conn(0x7f7e2c1430c0 legacy=0x7f7e2c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:05.416 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 1 monclient: found mon.b 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c1d8340 con 0x7f7e2c13f710 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 10 monclient: _finish_auth 0 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 10 monclient: _renew_subs 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.76:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c134820 con 0x7f7e2c13f710 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 10 monclient: _renew_subs 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:52:05.417 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.76:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c1380c0 con 0x7f7e2c13f710 2024-03-25T19:52:05.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.415+0000 7f7e30857640 1 librados: init done 2024-03-25T19:52:05.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.416+0000 7f7e30857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.416+0000 7f7e30857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.416+0000 7f7e30857640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:52:05.418 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.416+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c0627b0 con 0x7f7e2c13f710 2024-03-25T19:52:05.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.689+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e140038a0 con 0x7f7e2c13f710 2024-03-25T19:52:05.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.689+0000 7f7e22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:05.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.689+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7e14003bb0 con 0x7f7e2c13f710 2024-03-25T19:52:05.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.689+0000 7f7e22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.689+0000 7f7e22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:05.690 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.689+0000 7f7e22ffd640 10 monclient: dump: 2024-03-25T19:52:05.690 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:05.691 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.689+0000 7f7e2959a640 4 set_mon_vals no callback set 2024-03-25T19:52:05.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.690+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f7e14019b80 con 0x7f7e2c13f710 2024-03-25T19:52:05.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.690+0000 7f7e2a59c640 10 monclient: get_auth_request con 0x7f7e0c06df50 auth_method 0 2024-03-25T19:52:05.692 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.691+0000 7f7e22ffd640 10 client.4726.objecter ms_handle_connect 0x7f7e0c06df50 2024-03-25T19:52:05.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.695+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7f7e14053660 con 0x7f7e2c13f710 2024-03-25T19:52:05.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.695+0000 7f7e22ffd640 10 client.4726.objecter ms_dispatch 0x7f7e2c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:05.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.695+0000 7f7e22ffd640 3 client.4726.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:05.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.695+0000 7f7e22ffd640 3 client.4726.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:05.696 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.695+0000 7f7e22ffd640 20 client.4726.objecter dump_active .. 0 homeless 2024-03-25T19:52:05.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.696+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e1404f170 con 0x7f7e2c13f710 2024-03-25T19:52:05.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.696+0000 7f7e22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.697 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.696+0000 7f7e22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:05.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.948+0000 7f7e30857640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.949+0000 7f7e30857640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.949 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.949+0000 7f7e30857640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.76:6789/0 2024-03-25T19:52:05.950 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:05.949+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 --> v1:172.21.15.76:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e2c14e8b0 con 0x7f7e2c13f710 2024-03-25T19:52:06.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.080+0000 7f7e22ffd640 1 -- 172.21.15.50:0/150165049 <== mon.1 v1:172.21.15.76:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7f7e140510a0 con 0x7f7e2c13f710 2024-03-25T19:52:06.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.080+0000 7f7e22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:06.081 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.080+0000 7f7e22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:06.082 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:52:06.082 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 10 librados: watch_flush enter 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 10 librados: watch_flush exit 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 20 client.4726.objecter shutdown clearing up homeless session... 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 10 client.4726.objecter successfully canceled tick 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 >> v1:172.21.15.76:6800/2572556811 conn(0x7f7e0c06df50 legacy=0x7f7e0c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 10 monclient: shutdown 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 >> v1:172.21.15.76:6789/0 conn(0x7f7e2c13f710 legacy=0x7f7e2c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:06.084 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 shutdown_connections 2024-03-25T19:52:06.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 >> 172.21.15.50:0/150165049 conn(0x7f7e2c0b7ea0 msgr2=0x7f7e2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:06.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 shutdown_connections 2024-03-25T19:52:06.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 1 -- 172.21.15.50:0/150165049 wait complete. 2024-03-25T19:52:06.085 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:06.083+0000 7f7e30857640 1 librados: shutdown 2024-03-25T19:52:06.095 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:06.452 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:26.958 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.470 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:26.976 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.504 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:27.010 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.530 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:27.000 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.533 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:27.003 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.535 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:27.041 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.545 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:27.050 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.560 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:27.031 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:06.577 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:27.082 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:12.901 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.011+0000 7f673c4c8640 10 monclient: get_monmap_and_config 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.011+0000 7f673c4c8640 10 monclient: build_initial_monmap 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.011+0000 7f673c4c8640 10 monclient: monmap: 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:52:13.012366+0000 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:52:13.012366+0000 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:52:13.011 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:13.012 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 1 Processor -- start 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 1 -- start start 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient: init 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient(hunting): picked mon.noname-a con 0x7f67340a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient(hunting): picked mon.noname-b con 0x7f673413f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient(hunting): picked mon.noname-c con 0x7f67341430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6734142520 con 0x7f67340a98f0 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340b28a0 con 0x7f67341430c0 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340b2a30 con 0x7f673413f710 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.012+0000 7f673c4c8640 10 monclient(hunting): authenticate will time out at 5508.276580s 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.013+0000 7f673aa65640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f67341430c0 0x7f673414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:47206/0 (socket says 172.21.15.50:47206) 2024-03-25T19:52:13.013 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.013+0000 7f673aa65640 1 -- 172.21.15.50:0/3208854603 learned_addr learned my addr 172.21.15.50:0/3208854603 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:13.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.093+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524766091 0 0) 0x7f6734142520 con 0x7f67340a98f0 2024-03-25T19:52:13.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.093+0000 7f6739262640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.093+0000 7f6739262640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.093+0000 7f6739262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.093+0000 7f6739262640 10 monclient(hunting): my global_id is 4752 2024-03-25T19:52:13.093 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.093+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6720003520 con 0x7f67340a98f0 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2545130927 0 0) 0x7f6720003520 con 0x7f67340a98f0 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 >> v1:172.21.15.50:6790/0 conn(0x7f673413f710 legacy=0x7f67341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 >> v1:172.21.15.76:6789/0 conn(0x7f67341430c0 legacy=0x7f673414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 10 monclient: _finish_hunting 0 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 1 monclient: found mon.noname-a 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.50:6789/0 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340b44c0 con 0x7f67340a98f0 2024-03-25T19:52:13.113 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 10 monclient: _finish_auth 0 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 10 monclient: _check_auth_tickets 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6730001690 con 0x7f67340a98f0 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f673c4c8640 5 monclient: authenticate success, global_id 4752 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f673c4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f6739262640 10 monclient: dump: 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:13.114 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:13.115 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:13.115 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:13.115 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:13.115 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.113+0000 7f673c4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 <== mon.0 v1:172.21.15.50:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67300017f0 con 0x7f67340a98f0 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f6739262640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f6739262640 1 -- 172.21.15.50:0/3208854603 <== mon.0 v1:172.21.15.50:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f6730001c00 con 0x7f67340a98f0 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f6739262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f6739262640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f6739262640 10 monclient: dump: 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:13.151 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:13.152 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:13.152 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:13.152 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:13.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f673c4c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:13.152 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.151+0000 7f673c4c8640 4 set_mon_vals no callback set 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.152+0000 7f673c4c8640 10 monclient: shutdown 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.152+0000 7f673c4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.152+0000 7f673c4c8640 1 -- 172.21.15.50:0/3208854603 >> v1:172.21.15.50:6789/0 conn(0x7f67340a98f0 legacy=0x7f67340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.152+0000 7f673c4c8640 1 -- 172.21.15.50:0/3208854603 shutdown_connections 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.152+0000 7f673c4c8640 1 -- 172.21.15.50:0/3208854603 >> 172.21.15.50:0/3208854603 conn(0x7f67340b7ea0 msgr2=0x7f67340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.152+0000 7f673c4c8640 1 -- 172.21.15.50:0/3208854603 shutdown_connections 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.152+0000 7f673c4c8640 1 -- 172.21.15.50:0/3208854603 wait complete. 2024-03-25T19:52:13.153 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 10 monclient: build_initial_monmap 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 10 monclient: monmap: 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 1 librados: starting msgr at 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 1 librados: starting objecter 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 1 Processor -- start 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 1 -- start start 2024-03-25T19:52:13.154 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 1 librados: setting wanted keys 2024-03-25T19:52:13.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 1 librados: calling monclient init 2024-03-25T19:52:13.155 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.153+0000 7f673c4c8640 10 monclient: init 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 10 monclient(hunting): picked mon.noname-b con 0x7f67340a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 10 monclient(hunting): picked mon.noname-c con 0x7f673413f710 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 10 monclient(hunting): picked mon.noname-a con 0x7f67341430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6734135ac0 con 0x7f67341430c0 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6734137030 con 0x7f67340a98f0 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67341372a0 con 0x7f673413f710 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.154+0000 7f673c4c8640 10 monclient(hunting): authenticate will time out at 5508.418436s 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.156+0000 7f673a264640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f67340a98f0 0x7f6734133440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:47238/0 (socket says 172.21.15.50:47238) 2024-03-25T19:52:13.156 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.156+0000 7f673a264640 1 -- 172.21.15.50:0/2382776072 learned_addr learned my addr 172.21.15.50:0/2382776072 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:13.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.156+0000 7f672affd640 10 client.?.objecter ms_handle_connect 0x7f67340a98f0 2024-03-25T19:52:13.157 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.157+0000 7f672affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:13.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.158+0000 7f672affd640 10 client.?.objecter ms_handle_connect 0x7f673413f710 2024-03-25T19:52:13.158 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.158+0000 7f672affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:13.172 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.173+0000 7f672affd640 10 client.?.objecter ms_handle_connect 0x7f67341430c0 2024-03-25T19:52:13.172 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.173+0000 7f672affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.228+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541310856 0 0) 0x7f6734135ac0 con 0x7f67341430c0 2024-03-25T19:52:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.228+0000 7f672affd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.228+0000 7f672affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.228+0000 7f672affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.228+0000 7f672affd640 10 monclient(hunting): my global_id is 4755 2024-03-25T19:52:13.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.228+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6718004eb0 con 0x7f67341430c0 2024-03-25T19:52:13.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.240+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326964598 0 0) 0x7f6734137030 con 0x7f67340a98f0 2024-03-25T19:52:13.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.240+0000 7f672affd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.240+0000 7f672affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.240+0000 7f672affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.240+0000 7f672affd640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:52:13.240 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.240+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6734135ac0 con 0x7f67340a98f0 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.241+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 898731522 0 0) 0x7f6718004eb0 con 0x7f67341430c0 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.241+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6734137030 con 0x7f67341430c0 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.241+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f672c001b00 con 0x7f67341430c0 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.241+0000 7f672affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.241+0000 7f672affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.241+0000 7f672affd640 10 monclient(hunting): dump: 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.241 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.242 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.242 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:13.242 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:13.242 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:13.242 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:13.242 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:13.242 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.241+0000 7f672affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.249+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845739862 0 0) 0x7f67341372a0 con 0x7f673413f710 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.249+0000 7f672affd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.249+0000 7f672affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.249+0000 7f672affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.249+0000 7f672affd640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:52:13.249 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.249+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6718004eb0 con 0x7f673413f710 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 552939515 0 0) 0x7f6734137030 con 0x7f67341430c0 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 >> v1:172.21.15.50:6790/0 conn(0x7f673413f710 legacy=0x7f67341379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 >> v1:172.21.15.76:6789/0 conn(0x7f67340a98f0 legacy=0x7f6734133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 1 monclient: found mon.a 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734137580 con 0x7f67341430c0 2024-03-25T19:52:13.251 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 10 monclient: _finish_auth 0 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f672affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 5 monclient: authenticate success, global_id 4755 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 10 monclient: _renew_subs 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67341363b0 con 0x7f67341430c0 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 10 client.4755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 10 monclient: _renew_subs 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67341353a0 con 0x7f67341430c0 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.250+0000 7f673c4c8640 1 librados: init done 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.251+0000 7f673c4c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:13.252 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.251+0000 7f673c4c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:13.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.251+0000 7f673c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:13.253 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.251+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67340627b0 con 0x7f67341430c0 2024-03-25T19:52:13.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.585+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f672c001f70 con 0x7f67341430c0 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.585+0000 7f672affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.585+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f672c004d00 con 0x7f67341430c0 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.585+0000 7f672affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.585+0000 7f672affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.585+0000 7f672affd640 10 monclient: dump: 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:13.586 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:13.587 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:13.587 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:13.587 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:13.587 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:13.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.585+0000 7f6738a61640 4 set_mon_vals no callback set 2024-03-25T19:52:13.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.586+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f672c004f20 con 0x7f67341430c0 2024-03-25T19:52:13.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.587+0000 7f673a264640 10 monclient: get_auth_request con 0x7f671806de00 auth_method 0 2024-03-25T19:52:13.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.587+0000 7f672affd640 10 client.4755.objecter ms_handle_connect 0x7f671806de00 2024-03-25T19:52:13.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7f672c054960 con 0x7f67341430c0 2024-03-25T19:52:13.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 10 client.4755.objecter ms_dispatch 0x7f6734000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:13.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 3 client.4755.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:13.593 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 3 client.4755.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:13.593 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 20 client.4755.objecter dump_active .. 0 homeless 2024-03-25T19:52:13.593 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f672c050470 con 0x7f67341430c0 2024-03-25T19:52:13.593 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:13.593 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.592+0000 7f672affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:13.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.851+0000 7f673c4c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.851+0000 7f673c4c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.851+0000 7f673c4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:13.851 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.851+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f673413e1f0 con 0x7f67341430c0 2024-03-25T19:52:13.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.880+0000 7f672affd640 1 -- 172.21.15.50:0/2382776072 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7f672c0523a0 con 0x7f67341430c0 2024-03-25T19:52:13.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.880+0000 7f672affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.880 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.880+0000 7f672affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:13.880 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:52:13.881 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.882+0000 7f673c4c8640 10 librados: watch_flush enter 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.882+0000 7f673c4c8640 10 librados: watch_flush exit 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.882+0000 7f673c4c8640 20 client.4755.objecter shutdown clearing up homeless session... 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.882+0000 7f673c4c8640 10 client.4755.objecter successfully canceled tick 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 >> v1:172.21.15.76:6800/2572556811 conn(0x7f671806de00 legacy=0x7f67180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 10 monclient: shutdown 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 >> v1:172.21.15.50:6789/0 conn(0x7f67341430c0 legacy=0x7f67341e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 shutdown_connections 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 >> 172.21.15.50:0/2382776072 conn(0x7f67340b7ea0 msgr2=0x7f6734074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 shutdown_connections 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 1 -- 172.21.15.50:0/2382776072 wait complete. 2024-03-25T19:52:13.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:13.883+0000 7f673c4c8640 1 librados: shutdown 2024-03-25T19:52:13.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:20.701 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:20.816 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.817+0000 7ff818f7b640 10 monclient: get_monmap_and_config 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.817+0000 7ff818f7b640 10 monclient: build_initial_monmap 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.817+0000 7ff818f7b640 10 monclient: monmap: 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:52:20.818013+0000 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:52:20.818013+0000 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:52:20.817 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:52:20.818 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.817+0000 7ff818f7b640 1 Processor -- start 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.817+0000 7ff818f7b640 1 -- start start 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.817+0000 7ff818f7b640 10 monclient: init 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 10 monclient(hunting): picked mon.noname-c con 0x7ff8140a98f0 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 10 monclient(hunting): picked mon.noname-a con 0x7ff81413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 10 monclient(hunting): picked mon.noname-b con 0x7ff8141430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff814142520 con 0x7ff81413f710 2024-03-25T19:52:20.819 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8140b28a0 con 0x7ff8140a98f0 2024-03-25T19:52:20.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8140b2a30 con 0x7ff8141430c0 2024-03-25T19:52:20.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:20.820 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.818+0000 7ff818f7b640 10 monclient(hunting): authenticate will time out at 5516.082297s 2024-03-25T19:52:20.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.829+0000 7ff809d9b640 1 --1- >> v1:172.21.15.50:6789/0 conn(0x7ff81413f710 0x7ff8141429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.50:6789/0 says I am v1:172.21.15.50:58068/0 (socket says 172.21.15.50:58068) 2024-03-25T19:52:20.829 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.829+0000 7ff809d9b640 1 -- 172.21.15.50:0/3660408090 learned_addr learned my addr 172.21.15.50:0/3660408090 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:20.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.846+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750065174 0 0) 0x7ff8140b28a0 con 0x7ff8140a98f0 2024-03-25T19:52:20.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.846+0000 7ff811d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.846+0000 7ff811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.846+0000 7ff811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.846+0000 7ff811d9b640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:52:20.846 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.846+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff800003520 con 0x7ff8140a98f0 2024-03-25T19:52:20.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.857+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779276844 0 0) 0x7ff8140b2a30 con 0x7ff8141430c0 2024-03-25T19:52:20.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.857+0000 7ff811d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.857+0000 7ff811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.857+0000 7ff811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.857+0000 7ff811d9b640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:52:20.857 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.857+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8140b28a0 con 0x7ff8141430c0 2024-03-25T19:52:20.858 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.859+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062721334 0 0) 0x7ff814142520 con 0x7ff81413f710 2024-03-25T19:52:20.858 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.859+0000 7ff811d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.859+0000 7ff811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.859+0000 7ff811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.859+0000 7ff811d9b640 10 monclient(hunting): my global_id is 4758 2024-03-25T19:52:20.859 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.859+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8140b2a30 con 0x7ff81413f710 2024-03-25T19:52:20.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158951974 0 0) 0x7ff800003520 con 0x7ff8140a98f0 2024-03-25T19:52:20.874 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 >> v1:172.21.15.50:6790/0 conn(0x7ff8141430c0 legacy=0x7ff81414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 >> v1:172.21.15.50:6789/0 conn(0x7ff81413f710 legacy=0x7ff8141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 1 monclient: found mon.noname-c 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.76:6789/0 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 --> v1:172.21.15.76:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8140b44c0 con 0x7ff8140a98f0 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 10 monclient: _finish_auth 0 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff804001690 con 0x7ff8140a98f0 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff818f7b640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff818f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.875 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff811d9b640 10 monclient: dump: 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:20.876 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.874+0000 7ff818f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:20.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 <== mon.1 v1:172.21.15.76:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8040017f0 con 0x7ff8140a98f0 2024-03-25T19:52:20.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff811d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:20.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff811d9b640 1 -- 172.21.15.50:0/3660408090 <== mon.1 v1:172.21.15.76:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff804001c00 con 0x7ff8140a98f0 2024-03-25T19:52:20.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff811d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff811d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:20.883 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff811d9b640 10 monclient: dump: 2024-03-25T19:52:20.883 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff818f7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:20.884 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.883+0000 7ff818f7b640 4 set_mon_vals no callback set 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.884+0000 7ff818f7b640 10 monclient: shutdown 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.884+0000 7ff818f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.884+0000 7ff818f7b640 1 -- 172.21.15.50:0/3660408090 >> v1:172.21.15.76:6789/0 conn(0x7ff8140a98f0 legacy=0x7ff8140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.884+0000 7ff818f7b640 1 -- 172.21.15.50:0/3660408090 shutdown_connections 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.884+0000 7ff818f7b640 1 -- 172.21.15.50:0/3660408090 >> 172.21.15.50:0/3660408090 conn(0x7ff8140b7ea0 msgr2=0x7ff8140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.884+0000 7ff818f7b640 1 -- 172.21.15.50:0/3660408090 shutdown_connections 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.885+0000 7ff818f7b640 1 -- 172.21.15.50:0/3660408090 wait complete. 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.885+0000 7ff818f7b640 10 monclient: build_initial_monmap 2024-03-25T19:52:20.886 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.885+0000 7ff818f7b640 10 monclient: monmap: 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.886+0000 7ff818f7b640 1 librados: starting msgr at 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.886+0000 7ff818f7b640 1 librados: starting objecter 2024-03-25T19:52:20.887 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.886+0000 7ff818f7b640 1 Processor -- start 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.886+0000 7ff818f7b640 1 -- start start 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.886+0000 7ff818f7b640 1 librados: setting wanted keys 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.886+0000 7ff818f7b640 1 librados: calling monclient init 2024-03-25T19:52:20.888 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.886+0000 7ff818f7b640 10 monclient: init 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 10 monclient(hunting): picked mon.noname-a con 0x7ff8140a98f0 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 10 monclient(hunting): picked mon.noname-b con 0x7ff81413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 10 monclient(hunting): picked mon.noname-c con 0x7ff8141430c0 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff814137ab0 con 0x7ff8140a98f0 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff814138bd0 con 0x7ff81413f710 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff814138e40 con 0x7ff8141430c0 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.887+0000 7ff818f7b640 10 monclient(hunting): authenticate will time out at 5516.151169s 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.889+0000 7ff812d9d640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7ff8141430c0 0x7ff8141e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:44264/0 (socket says 172.21.15.50:44264) 2024-03-25T19:52:20.890 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.889+0000 7ff812d9d640 1 -- 172.21.15.50:0/1976799345 learned_addr learned my addr 172.21.15.50:0/1976799345 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:20.891 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.890+0000 7ff80b7fe640 10 client.?.objecter ms_handle_connect 0x7ff81413f710 2024-03-25T19:52:20.891 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.890+0000 7ff80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:20.891 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.890+0000 7ff80b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8141430c0 2024-03-25T19:52:20.891 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.890+0000 7ff80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:20.941 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.942+0000 7ff80b7fe640 10 client.?.objecter ms_handle_connect 0x7ff8140a98f0 2024-03-25T19:52:20.942 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.942+0000 7ff80b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.948+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018345135 0 0) 0x7ff814137ab0 con 0x7ff8140a98f0 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.948+0000 7ff80b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.948+0000 7ff80b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.948+0000 7ff80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.948+0000 7ff80b7fe640 10 monclient(hunting): my global_id is 4761 2024-03-25T19:52:20.948 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.948+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f4004eb0 con 0x7ff8140a98f0 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.961+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2737572148 0 0) 0x7ff7f4004eb0 con 0x7ff8140a98f0 2024-03-25T19:52:20.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.962+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 --> v1:172.21.15.50:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff814137ab0 con 0x7ff8140a98f0 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.962+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff804001b00 con 0x7ff8140a98f0 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.962+0000 7ff80b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.962+0000 7ff80b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.962+0000 7ff80b7fe640 10 monclient(hunting): dump: 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:20.962 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:20.963 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:20.963 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:20.963 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.962+0000 7ff80b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 43093508 0 0) 0x7ff814137ab0 con 0x7ff8140a98f0 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 >> v1:172.21.15.50:6790/0 conn(0x7ff8141430c0 legacy=0x7ff8141e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 >> v1:172.21.15.76:6789/0 conn(0x7ff81413f710 legacy=0x7ff8141df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 1 monclient: found mon.a 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 --> v1:172.21.15.50:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff814139120 con 0x7ff8140a98f0 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff80b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 5 monclient: authenticate success, global_id 4761 2024-03-25T19:52:20.983 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 10 monclient: _renew_subs 2024-03-25T19:52:20.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:20.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 --> v1:172.21.15.50:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff814139400 con 0x7ff8140a98f0 2024-03-25T19:52:20.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 10 client.4761.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:20.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 10 monclient: _renew_subs 2024-03-25T19:52:20.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:20.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 --> v1:172.21.15.50:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8140b7880 con 0x7ff8140a98f0 2024-03-25T19:52:20.984 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.982+0000 7ff818f7b640 1 librados: init done 2024-03-25T19:52:20.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.984+0000 7ff818f7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:20.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.984+0000 7ff818f7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:20.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.984+0000 7ff818f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:20.985 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:20.984+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff814137810 con 0x7ff8140a98f0 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.277+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff804001220 con 0x7ff8140a98f0 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.277+0000 7ff80b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.277+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7ff804003a40 con 0x7ff8140a98f0 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.277+0000 7ff80b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.277+0000 7ff80b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.277+0000 7ff80b7fe640 10 monclient: dump: 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:21.277 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:21.278 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:21.278 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:21.278 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:21.278 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:21.278 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:21.278 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:21.278 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.277+0000 7ff81159a640 4 set_mon_vals no callback set 2024-03-25T19:52:21.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.278+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7ff804019a50 con 0x7ff8140a98f0 2024-03-25T19:52:21.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.278+0000 7ff809d9b640 10 monclient: get_auth_request con 0x7ff7f406dc00 auth_method 0 2024-03-25T19:52:21.279 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.279+0000 7ff80b7fe640 10 client.4761.objecter ms_handle_connect 0x7ff7f406dc00 2024-03-25T19:52:21.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7ff804053700 con 0x7ff8140a98f0 2024-03-25T19:52:21.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 10 client.4761.objecter ms_dispatch 0x7ff814000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:21.286 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 3 client.4761.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:21.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 3 client.4761.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:21.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 20 client.4761.objecter dump_active .. 0 homeless 2024-03-25T19:52:21.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff80404f040 con 0x7ff8140a98f0 2024-03-25T19:52:21.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:21.287 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.287+0000 7ff80b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:21.546 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.546+0000 7ff818f7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:21.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.546+0000 7ff818f7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:21.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.546+0000 7ff818f7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.50:6789/0 2024-03-25T19:52:21.547 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.546+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 --> v1:172.21.15.50:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8140627b0 con 0x7ff8140a98f0 2024-03-25T19:52:21.581 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.581+0000 7ff80b7fe640 1 -- 172.21.15.50:0/1976799345 <== mon.0 v1:172.21.15.50:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7ff804051140 con 0x7ff8140a98f0 2024-03-25T19:52:21.581 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.581+0000 7ff80b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:21.581 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.581+0000 7ff80b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:21.582 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:52:21.582 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:21.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 10 librados: watch_flush enter 2024-03-25T19:52:21.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 10 librados: watch_flush exit 2024-03-25T19:52:21.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 20 client.4761.objecter shutdown clearing up homeless session... 2024-03-25T19:52:21.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 10 client.4761.objecter successfully canceled tick 2024-03-25T19:52:21.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 >> v1:172.21.15.76:6800/2572556811 conn(0x7ff7f406dc00 legacy=0x7ff7f40700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:21.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 10 monclient: shutdown 2024-03-25T19:52:21.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:21.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 >> v1:172.21.15.50:6789/0 conn(0x7ff8140a98f0 legacy=0x7ff814137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:21.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 shutdown_connections 2024-03-25T19:52:21.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 >> 172.21.15.50:0/1976799345 conn(0x7ff8140b7ea0 msgr2=0x7ff814074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:21.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 shutdown_connections 2024-03-25T19:52:21.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 1 -- 172.21.15.50:0/1976799345 wait complete. 2024-03-25T19:52:21.585 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:21.584+0000 7ff818f7b640 1 librados: shutdown 2024-03-25T19:52:21.597 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:26.516 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:47.022 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.524 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:47.030 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.570 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:47.041 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.575 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:11:47.081 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.584 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:47.089 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.584 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:47.055 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.590 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:47.096 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.611 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:11:47.082 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:26.620 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:11:47.126 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.404 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.519+0000 7f7588f89640 10 monclient: get_monmap_and_config 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.519+0000 7f7588f89640 10 monclient: build_initial_monmap 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.519+0000 7f7588f89640 10 monclient: monmap: 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:52:28.520333+0000 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:52:28.520333+0000 2024-03-25T19:52:28.519 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:28.520 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:28.520 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:28.520 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-c 2024-03-25T19:52:28.520 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-b 2024-03-25T19:52:28.520 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:28.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.520+0000 7f7588f89640 1 Processor -- start 2024-03-25T19:52:28.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.520+0000 7f7588f89640 1 -- start start 2024-03-25T19:52:28.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.520+0000 7f7588f89640 10 monclient: init 2024-03-25T19:52:28.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:28.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:52:28.521 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 10 monclient(hunting): picked mon.noname-b con 0x7f75840a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:28.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 10 monclient(hunting): picked mon.noname-c con 0x7f758413f710 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:28.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 10 monclient(hunting): picked mon.noname-a con 0x7f75841430c0 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:28.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7584142520 con 0x7f75841430c0 2024-03-25T19:52:28.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840b28a0 con 0x7f758413f710 2024-03-25T19:52:28.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75840b2a30 con 0x7f75840a98f0 2024-03-25T19:52:28.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:28.522 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.521+0000 7f7588f89640 10 monclient(hunting): authenticate will time out at 5523.785068s 2024-03-25T19:52:28.528 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.528+0000 7f7581d9b640 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f758413f710 0x7f75841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.50:58324/0 (socket says 172.21.15.50:58324) 2024-03-25T19:52:28.528 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.528+0000 7f7581d9b640 1 -- 172.21.15.50:0/2625974489 learned_addr learned my addr 172.21.15.50:0/2625974489 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:28.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.539+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054843418 0 0) 0x7f75840b2a30 con 0x7f75840a98f0 2024-03-25T19:52:28.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.539+0000 7f758159a640 10 monclient(hunting): _negotiate 2024-03-25T19:52:28.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.539+0000 7f758159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:28.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.539+0000 7f758159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:28.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.539+0000 7f758159a640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:52:28.539 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.539+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c003520 con 0x7f75840a98f0 2024-03-25T19:52:28.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.543+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709035716 0 0) 0x7f7584142520 con 0x7f75841430c0 2024-03-25T19:52:28.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.543+0000 7f758159a640 10 monclient(hunting): _negotiate 2024-03-25T19:52:28.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.543+0000 7f758159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:28.542 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.543+0000 7f758159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:28.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.543+0000 7f758159a640 10 monclient(hunting): my global_id is 4764 2024-03-25T19:52:28.543 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.543+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75840b2a30 con 0x7f75841430c0 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.563+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126449515 0 0) 0x7f75840b28a0 con 0x7f758413f710 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.563+0000 7f758159a640 10 monclient(hunting): _negotiate 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.563+0000 7f758159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.563+0000 7f758159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.563+0000 7f758159a640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.563+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584142520 con 0x7f758413f710 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1118269322 0 0) 0x7f756c003520 con 0x7f75840a98f0 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 >> v1:172.21.15.76:6789/0 conn(0x7f758413f710 legacy=0x7f75841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 >> v1:172.21.15.50:6789/0 conn(0x7f75841430c0 legacy=0x7f758414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 10 monclient: _finish_hunting 0 2024-03-25T19:52:28.564 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 1 monclient: found mon.noname-b 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.50:6790/0 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840b44c0 con 0x7f75840a98f0 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 10 monclient: _finish_auth 0 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 10 monclient: _check_auth_tickets 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7574001690 con 0x7f75840a98f0 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f7588f89640 5 monclient: authenticate success, global_id 4721 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f7588f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f758159a640 10 monclient: dump: 2024-03-25T19:52:28.565 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:28.566 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.564+0000 7f7588f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 <== mon.2 v1:172.21.15.50:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75740017f0 con 0x7f75840a98f0 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f758159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f758159a640 1 -- 172.21.15.50:0/2625974489 <== mon.2 v1:172.21.15.50:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7574001c00 con 0x7f75840a98f0 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f758159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f758159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f758159a640 10 monclient: dump: 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.583 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f7588f89640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:28.584 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.583+0000 7f7588f89640 4 set_mon_vals no callback set 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.584+0000 7f7588f89640 10 monclient: shutdown 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.584+0000 7f7588f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.584+0000 7f7588f89640 1 -- 172.21.15.50:0/2625974489 >> v1:172.21.15.50:6790/0 conn(0x7f75840a98f0 legacy=0x7f75840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.584+0000 7f7588f89640 1 -- 172.21.15.50:0/2625974489 shutdown_connections 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.584+0000 7f7588f89640 1 -- 172.21.15.50:0/2625974489 >> 172.21.15.50:0/2625974489 conn(0x7f75840b7ea0 msgr2=0x7f75840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.584+0000 7f7588f89640 1 -- 172.21.15.50:0/2625974489 shutdown_connections 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.584+0000 7f7588f89640 1 -- 172.21.15.50:0/2625974489 wait complete. 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 10 monclient: build_initial_monmap 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 10 monclient: monmap: 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:epoch 0 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:last_changed 0.000000 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:created 0.000000 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 1 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.noname-a 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.noname-b 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.noname-c 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 1 librados: starting msgr at 2024-03-25T19:52:28.587 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 1 librados: starting objecter 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 1 Processor -- start 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 1 -- start start 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 1 librados: setting wanted keys 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 1 librados: calling monclient init 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.585+0000 7f7588f89640 10 monclient: init 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 10 monclient(hunting): picked mon.noname-c con 0x7f75840a98f0 addr v1:172.21.15.50:6790/0 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 10 monclient(hunting): picked mon.noname-a con 0x7f758413f710 addr v1:172.21.15.50:6789/0 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 10 monclient(hunting): picked mon.noname-b con 0x7f75841430c0 addr v1:172.21.15.76:6789/0 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 1 -- --> v1:172.21.15.50:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7584138b20 con 0x7f758413f710 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7584137a80 con 0x7f75841430c0 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 1 -- --> v1:172.21.15.50:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7584139880 con 0x7f75840a98f0 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:28.588 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.586+0000 7f7588f89640 10 monclient(hunting): authenticate will time out at 5523.850133s 2024-03-25T19:52:28.591 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.591+0000 7f758259c640 1 --1- >> v1:172.21.15.50:6790/0 conn(0x7f75840a98f0 0x7f7584137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.50:6790/0 says I am v1:172.21.15.50:60562/0 (socket says 172.21.15.50:60562) 2024-03-25T19:52:28.591 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.591+0000 7f758259c640 1 -- 172.21.15.50:0/1079546268 learned_addr learned my addr 172.21.15.50:0/1079546268 (peer_addr_for_me v1:172.21.15.50:0/0) 2024-03-25T19:52:28.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.592+0000 7f7572ffd640 10 client.?.objecter ms_handle_connect 0x7f75840a98f0 2024-03-25T19:52:28.592 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.592+0000 7f7572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:28.602 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.603+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907711809 0 0) 0x7f7584139880 con 0x7f75840a98f0 2024-03-25T19:52:28.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.603+0000 7f7572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:28.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.603+0000 7f7572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:28.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.603+0000 7f7572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:28.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.603+0000 7f7572ffd640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:52:28.603 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.603+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7564004eb0 con 0x7f75840a98f0 2024-03-25T19:52:28.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.605+0000 7f7572ffd640 10 client.?.objecter ms_handle_connect 0x7f75841430c0 2024-03-25T19:52:28.604 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.605+0000 7f7572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:28.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.613+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587064623 0 0) 0x7f7584137a80 con 0x7f75841430c0 2024-03-25T19:52:28.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.613+0000 7f7572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:28.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.613+0000 7f7572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:28.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.613+0000 7f7572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:28.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.613+0000 7f7572ffd640 10 monclient(hunting): my global_id is 4744 2024-03-25T19:52:28.612 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.613+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584139880 con 0x7f75841430c0 2024-03-25T19:52:28.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.617+0000 7f7572ffd640 10 client.?.objecter ms_handle_connect 0x7f758413f710 2024-03-25T19:52:28.617 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.617+0000 7f7572ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.623+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425453480 0 0) 0x7f7564004eb0 con 0x7f75840a98f0 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.623+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7584137a80 con 0x7f75840a98f0 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.623+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7574001bc0 con 0x7f75840a98f0 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.623+0000 7f7572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.623+0000 7f7572ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.623+0000 7f7572ffd640 10 monclient(hunting): dump: 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.623 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:28.624 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:28.624 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:28.624 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:28.624 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:28.624 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:28.624 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.623+0000 7f7572ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:28.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.628+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.0 v1:172.21.15.50:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054062846 0 0) 0x7f7584138b20 con 0x7f758413f710 2024-03-25T19:52:28.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.628+0000 7f7572ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:28.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.628+0000 7f7572ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:28.627 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.628+0000 7f7572ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:28.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.628+0000 7f7572ffd640 10 monclient(hunting): my global_id is 4767 2024-03-25T19:52:28.628 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.628+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7564004eb0 con 0x7f758413f710 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.635+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2201649170 0 0) 0x7f7584139880 con 0x7f75841430c0 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.635+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7584138b20 con 0x7f75841430c0 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.635+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f757c001bd0 con 0x7f75841430c0 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.635+0000 7f7572ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.635+0000 7f7572ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.635+0000 7f7572ffd640 10 monclient(hunting): dump: 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:28.635 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:28.636 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:28.636 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:28.636 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:28.636 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 494630313 0 0) 0x7f7584137a80 con 0x7f75840a98f0 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 >> v1:172.21.15.76:6789/0 conn(0x7f75841430c0 legacy=0x7f75841e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 >> v1:172.21.15.50:6789/0 conn(0x7f758413f710 legacy=0x7f7584139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 1 monclient: found mon.c 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584139a10 con 0x7f75840a98f0 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 10 monclient: _finish_auth 0 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 5 monclient: authenticate success, global_id 4724 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 10 monclient: _renew_subs 2024-03-25T19:52:28.642 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75841dab00 con 0x7f75840a98f0 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 10 monclient: _renew_subs 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f758413b4f0 con 0x7f75840a98f0 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.642+0000 7f7588f89640 1 librados: init done 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.643+0000 7f7588f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.643+0000 7f7588f89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:28.643 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.643+0000 7f7588f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:52:28.644 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.643+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75841dae00 con 0x7f75840a98f0 2024-03-25T19:52:28.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.958+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75740011f0 con 0x7f75840a98f0 2024-03-25T19:52:28.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.958+0000 7f7572ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:28.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.958+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 561637850 0 0) 0x7f7574003b00 con 0x7f75840a98f0 2024-03-25T19:52:28.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.958+0000 7f7572ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:28.958 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.958+0000 7f7572ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.958+0000 7f7572ffd640 10 monclient: dump: 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:epoch 1 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:fsid 2e1d843b-b6f3-4677-9562-2b2edda74f50 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:last_changed 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:created 2024-03-25T19:40:08.610141+0000 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:election_strategy: 3 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:0: v1:172.21.15.50:6789/0 mon.a 2024-03-25T19:52:28.959 INFO:teuthology.misc.health.smithi050.stderr:1: v1:172.21.15.76:6789/0 mon.b 2024-03-25T19:52:28.960 INFO:teuthology.misc.health.smithi050.stderr:2: v1:172.21.15.50:6790/0 mon.c 2024-03-25T19:52:28.960 INFO:teuthology.misc.health.smithi050.stderr: 2024-03-25T19:52:28.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.958+0000 7f7580d99640 4 set_mon_vals no callback set 2024-03-25T19:52:28.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.959+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2864363352 0 0) 0x7f7574003d20 con 0x7f75840a98f0 2024-03-25T19:52:28.960 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.960+0000 7f7581d9b640 10 monclient: get_auth_request con 0x7f756406e040 auth_method 0 2024-03-25T19:52:28.961 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.960+0000 7f7572ffd640 10 client.4724.objecter ms_handle_connect 0x7f756406e040 2024-03-25T19:52:28.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4087761760 0 0) 0x7f75740548d0 con 0x7f75840a98f0 2024-03-25T19:52:28.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 10 client.4724.objecter ms_dispatch 0x7f7584000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:28.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:28.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:28.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T19:52:28.965 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75740503e0 con 0x7f75840a98f0 2024-03-25T19:52:28.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:28.966 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:28.965+0000 7f7572ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:29.201 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.201+0000 7f7588f89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:29.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.202+0000 7f7588f89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:29.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.202+0000 7f7588f89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.50:6790/0 2024-03-25T19:52:29.202 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.202+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 --> v1:172.21.15.50:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75840b3e80 con 0x7f75840a98f0 2024-03-25T19:52:29.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.223+0000 7f7572ffd640 1 -- 172.21.15.50:0/1079546268 <== mon.2 v1:172.21.15.50:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 55092835) 0x7f7574052310 con 0x7f75840a98f0 2024-03-25T19:52:29.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.223+0000 7f7572ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:29.223 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.223+0000 7f7572ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:29.224 INFO:teuthology.misc.health.smithi050.stdout: 2024-03-25T19:52:29.224 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":11,"fsid":"2e1d843b-b6f3-4677-9562-2b2edda74f50","created":"2024-03-25T19:40:56.914941+0000","modified":"2024-03-25T19:51:01.081823+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:01.081823+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"227bd158-cf26-4f25-8ba8-aa8da6d6ba2d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6e59a3aa-99f7-4d22-b629-20d3efbb0038","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ff7c8b95-7815-449d-a015-a11978f86ec7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f9bee88b-b66f-48e7-ba74-a053b8686bdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0445de8-c8d1-4915-85bf-a033f063e3dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"115d18c4-4a9f-4476-be16-cb53ccb366b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7cc327fd-d9f4-4632-a4a5-551ac2414a09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f4c445e7-27c7-4718-a82d-16a303cb47b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 10 librados: watch_flush enter 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 10 librados: watch_flush exit 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 10 client.4724.objecter successfully canceled tick 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 >> v1:172.21.15.76:6800/2572556811 conn(0x7f756406e040 legacy=0x7f7564070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 10 monclient: shutdown 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 >> v1:172.21.15.50:6790/0 conn(0x7f75840a98f0 legacy=0x7f7584137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 shutdown_connections 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 >> 172.21.15.50:0/1079546268 conn(0x7f75840b7ea0 msgr2=0x7f7584074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 shutdown_connections 2024-03-25T19:52:29.227 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.226+0000 7f7588f89640 1 -- 172.21.15.50:0/1079546268 wait complete. 2024-03-25T19:52:29.228 INFO:teuthology.misc.health.smithi050.stderr:2024-03-25T19:52:29.227+0000 7f7588f89640 1 librados: shutdown 2024-03-25T19:52:29.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:29.240 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:52:29.241 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:52:29.242 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:52:29.243 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:31.309 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:11:28.852 35549== 2024-03-25T19:52:31.309 INFO:tasks.ceph.osd.0.smithi050.stderr:==00:00:11:28.852 35549== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:31.347 INFO:tasks.ceph.osd.0.smithi050.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:35.345 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:35.345 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:52:35.346 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:52:35.346 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:52:35.346 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:37.226 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:11:34.769 35550== 2024-03-25T19:52:37.227 INFO:tasks.ceph.osd.1.smithi050.stderr:==00:00:11:34.769 35550== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:37.270 INFO:tasks.ceph.osd.1.smithi050.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:41.448 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:52:41.448 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:52:41.449 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:52:41.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:43.406 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:11:40.931 35551== 2024-03-25T19:52:43.406 INFO:tasks.ceph.osd.2.smithi050.stderr:==00:00:11:40.931 35551== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:43.448 INFO:tasks.ceph.osd.2.smithi050.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:46.499 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:07.005 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.553 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:07.059 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.585 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:07.091 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.598 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:12:07.069 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.605 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:07.111 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.606 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:12:07.076 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.615 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:07.120 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.632 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:12:07.103 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:46.637 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:07.143 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:47.551 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:52:47.551 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:52:47.552 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:52:47.552 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:49.320 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:11:46.845 35552== 2024-03-25T19:52:49.321 INFO:tasks.ceph.osd.3.smithi050.stderr:==00:00:11:46.845 35552== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:49.365 INFO:tasks.ceph.osd.3.smithi050.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:53.654 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:52:53.654 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:52:53.655 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:52:53.655 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:52:55.728 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:11:53.272 35340== 2024-03-25T19:52:55.729 INFO:tasks.ceph.osd.4.smithi076.stderr:==00:00:11:53.272 35340== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:52:55.763 INFO:tasks.ceph.osd.4.smithi076.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:52:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:52:59.756 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:52:59.757 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:52:59.757 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:52:59.757 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:01.708 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:11:59.252 35341== 2024-03-25T19:53:01.708 INFO:tasks.ceph.osd.5.smithi076.stderr:==00:00:11:59.252 35341== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:01.750 INFO:tasks.ceph.osd.5.smithi076.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:05.859 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:53:05.860 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:53:05.860 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:53:05.860 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:06.531 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:27.037 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.578 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:27.084 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.595 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:27.101 35239== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.630 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:12:27.100 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.632 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:12:27.102 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.647 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:27.153 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.648 INFO:tasks.ceph.mon.b.smithi076.stderr:==00:00:12:27.119 34797== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.650 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:27.155 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.663 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:27.169 35240== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:07.729 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:12:05.270 35342== 2024-03-25T19:53:07.729 INFO:tasks.ceph.osd.6.smithi076.stderr:==00:00:12:05.270 35342== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:07.772 INFO:tasks.ceph.osd.6.smithi076.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:11.962 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:53:11.962 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:53:11.963 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:53:11.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:13.414 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:12:10.951 35343== 2024-03-25T19:53:13.414 INFO:tasks.ceph.osd.7.smithi076.stderr:==00:00:12:10.951 35343== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:13.449 INFO:tasks.ceph.osd.7.smithi076.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:18.065 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:53:18.065 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:53:18.066 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:53:18.066 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:53:18.066 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:18.092 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:53:18.092 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:53:18.092 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:18.119 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:53:18.119 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:53:18.119 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:53:18.119 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:18.148 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-03-25T19:53:18.137+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35203) UID: 0 2024-03-25T19:53:18.149 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-03-25T19:53:18.139+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:53:18.723 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:39.229 35239== 2024-03-25T19:53:18.723 INFO:tasks.ceph.mon.a.smithi050.stderr:==00:00:12:39.229 35239== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:18.733 INFO:tasks.ceph.mon.a.smithi050.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:24.221 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:53:24.221 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:53:24.221 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:53:24.222 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:24.240 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-03-25T19:53:24.227+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35205) UID: 0 2024-03-25T19:53:24.240 INFO:tasks.ceph.mon.c.smithi050.stderr:2024-03-25T19:53:24.228+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:53:24.918 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:45.424 35240== 2024-03-25T19:53:24.918 INFO:tasks.ceph.mon.c.smithi050.stderr:==00:00:12:45.424 35240== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:24.929 INFO:tasks.ceph.mon.c.smithi050.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:30.324 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:53:30.324 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:53:30.324 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:53:30.324 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:30.339 INFO:tasks.ceph.mon.b.smithi076.stderr:2024-03-25T19:53:30.329+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34793) UID: 0 2024-03-25T19:53:30.340 INFO:tasks.ceph.mon.b.smithi076.stderr:2024-03-25T19:53:30.331+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:58:29.398 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:58:29.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:58:29.446 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:58:29.447 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:58:29.505 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:58:29.506 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:58:29.563 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:58:29.563 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:58:29.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:58:29.730 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:58:29.789 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:58:29.789 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:58:29.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:58:29.841 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:58:29.891 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:58:29.891 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:58:29.940 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:58:29.940 DEBUG:teuthology.orchestra.run.smithi076:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:58:29.991 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:58:29.992 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/data/mon.a.tgz 2024-03-25T19:58:29.993 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-03-25T19:58:30.009 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.WznqaxFXjh 2024-03-25T19:58:30.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.WznqaxFXjh 2024-03-25T19:58:30.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.WznqaxFXjh 2024-03-25T19:58:30.228 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.WznqaxFXjh is 134KB 2024-03-25T19:58:30.280 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.WznqaxFXjh 2024-03-25T19:58:30.294 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/data/mon.c.tgz 2024-03-25T19:58:30.295 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-03-25T19:58:30.349 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.Rr44q5plHf 2024-03-25T19:58:30.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Rr44q5plHf 2024-03-25T19:58:30.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.Rr44q5plHf 2024-03-25T19:58:30.525 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.Rr44q5plHf is 135KB 2024-03-25T19:58:30.578 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.Rr44q5plHf 2024-03-25T19:58:30.592 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/data/mon.b.tgz 2024-03-25T19:58:30.592 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2024-03-25T19:58:30.607 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.h3SLDBh5qN 2024-03-25T19:58:30.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.h3SLDBh5qN 2024-03-25T19:58:30.712 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.h3SLDBh5qN 2024-03-25T19:58:30.797 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.h3SLDBh5qN is 135KB 2024-03-25T19:58:30.847 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.h3SLDBh5qN 2024-03-25T19:58:30.862 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:58:30.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:58:30.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:58:31.228 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:58:31.302 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:58:31.495 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:58:31.497 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/remote/smithi050/crash 2024-03-25T19:58:31.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:58:31.527 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/remote/smithi076/crash 2024-03-25T19:58:31.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:58:31.555 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:58:31.555 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:58:31.572 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:58:31.633 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:58:31.633 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/valgrind/osd.6.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34950.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr: 55.9% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/valgrind/osd.4.log: /var/log/ceph/ceph-client.admin.34950.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:58:31.634 INFO:teuthology.orchestra.run.smithi076.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34950.log.gz 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34990.log 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.35150.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.34990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35070.log 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2024-03-25T19:58:31.635 INFO:teuthology.orchestra.run.smithi076.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34990.log.gz 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35030.log 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.35070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35201.log 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr: 87.7%/var/log/ceph/ceph-client.admin.35030.log: -- replaced with /var/log/ceph/ceph-client.admin.35070.log.gz 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35241.log 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.35201.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35030.log.gz 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35201.log.gz 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.35241.log: -- /var/log/ceph/ceph-client.admin.35110.log 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35241.log.gz 2024-03-25T19:58:31.645 INFO:teuthology.orchestra.run.smithi076.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:58:31.652 INFO:teuthology.orchestra.run.smithi076.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:58:31.652 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/ceph-client.admin.35110.log: /var/log/ceph/ceph-mon.b.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35110.log.gz 2024-03-25T19:58:31.652 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40177.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39592.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40177.log.gz 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36672.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41777.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.36672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39904.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41231.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36672.log.gz 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39082.log 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41777.log.gz 2024-03-25T19:58:31.653 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41231.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39904.log.gz 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39082.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41231.log.gz 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41621.log 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42325.log 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41621.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42325.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41621.log.gz 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40216.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42325.log.gz 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-03-25T19:58:31.654 INFO:teuthology.orchestra.run.smithi050.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39358.log 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41543.log 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42364.log 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41543.log: gzip -5 --verbose -- 86.8% /var/log/ceph/ceph-client.admin.41699.log 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39358.log.gz 2024-03-25T19:58:31.655 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41818.log 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41543.log.gz 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40528.log 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42364.log.gz 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41699.log.gz 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41818.log.gz 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39943.log 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40528.log.gz 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39670.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42130.log 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39943.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2024-03-25T19:58:31.656 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41192.log 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39943.log.gz 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42130.log.gz 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41192.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40333.log 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41192.log.gz 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35245.log 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gz 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39121.log 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.35245.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35746.log 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40333.log.gz 2024-03-25T19:58:31.657 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40684.log 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35245.log.gz 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.35746.log: gzip -5 --verbose 86.2% -- -- replaced with /var/log/ceph/ceph-client.admin.39121.log.gz /var/log/ceph/ceph-client.admin.40958.log 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40138.log 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35746.log.gz 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40958.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40684.log.gz 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39865.log 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40021.log 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40958.log.gz 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40099.log 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40138.log.gz 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40021.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39865.log.gz 2024-03-25T19:58:31.658 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42091.log 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40099.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40021.log.gz 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41036.log 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40099.log.gz 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35575.log 86.3% 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42091.log.gz 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41857.log 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.35575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39280.log 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41036.log.gz 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41075.log 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35575.log.gz 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40489.log 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41857.log.gz 2024-03-25T19:58:31.659 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42247.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41075.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39280.log.gz 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41075.log.gz 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39202.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40489.log.gz 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40060.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42247.log.gz 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39826.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35548.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39202.log: /var/log/ceph/ceph-client.admin.39826.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.35548.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39202.log.gz 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39787.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39826.log.gz 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-03-25T19:58:31.660 INFO:teuthology.orchestra.run.smithi050.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35548.log.gz 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39475.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39787.log.gz 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42013.log 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39397.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz/var/log/ceph/ceph-client.admin.42013.log: 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40997.log 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.42013.log.gz 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29394.log 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42208.log 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.tmp-client.admin.29394.log: gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29394.log.gz 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40606.log 2024-03-25T19:58:31.661 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40997.log.gz 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41582.log 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40645.log 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42208.log.gz 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41270.log 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40606.log.gz 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39631.log 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40645.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41582.log.gz 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40645.log.gz 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39631.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41270.log.gz 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39436.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39631.log.gz 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37718.log 2024-03-25T19:58:31.662 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40450.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.37718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39319.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40450.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41387.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37718.log.gz 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41504.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40450.log.gz 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39241.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39319.log.gz 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41504.log: -- /var/log/ceph/ceph-client.admin.41348.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41387.log.gz 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41504.log.gz 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39241.log: /var/log/ceph/ceph-client.admin.41348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40880.log 2024-03-25T19:58:31.663 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41348.log.gz 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40723.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39241.log.gz 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz/var/log/ceph/ceph-client.admin.40880.log: 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41309.log 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr: 86.3%/var/log/ceph/ceph-client.admin.40802.log: -- replaced with /var/log/ceph/ceph-client.admin.40880.log.gz 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41309.log: /var/log/ceph/ceph-client.admin.40255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41896.log 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr: 86.8% 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41309.log.gz 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41896.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr: 86.6%/var/log/ceph/ceph-client.admin.39162.log: -- replaced with /var/log/ceph/ceph-client.admin.41896.log.gz 2024-03-25T19:58:31.664 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41153.log 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41974.log 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41738.log 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41153.log: /var/log/ceph/ceph-client.admin.41974.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41153.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41974.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41738.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41738.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42052.log 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42169.log: /var/log/ceph/ceph-client.admin.39514.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-25T19:58:31.665 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41426.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40411.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41426.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42052.log.gz 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39982.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41426.log.gz 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42286.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40411.log: /var/log/ceph/ceph-client.admin.39982.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40411.log.gz 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39982.log.gz 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40294.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41660.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.42286.log: /var/log/ceph/ceph-client.admin.40294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.41660.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42286.log.gz 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40294.log.gz 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41465.log 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41660.log.gz 2024-03-25T19:58:31.666 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40372.log 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.38873.log: /var/log/ceph/ceph-client.admin.41465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40841.log 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40372.log: 86.5% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41465.log.gz 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40372.log.gz 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40919.log 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.40841.log: /var/log/ceph/ceph-client.admin.40919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-mon.c.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40841.log.gz 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40919.log.gz 2024-03-25T19:58:31.667 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-client.admin.39748.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-03-25T19:58:31.706 INFO:teuthology.orchestra.run.smithi076.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:58:32.434 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:58:32.447 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:58:32.453 INFO:teuthology.orchestra.run.smithi076.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:58:32.550 INFO:teuthology.orchestra.run.smithi076.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:58:32.719 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:58:32.740 INFO:teuthology.orchestra.run.smithi050.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:58:32.793 INFO:teuthology.orchestra.run.smithi076.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:58:32.821 INFO:teuthology.orchestra.run.smithi076.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:58:36.732 INFO:teuthology.orchestra.run.smithi050.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:58:36.960 INFO:teuthology.orchestra.run.smithi050.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:58:36.962 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-03-25T19:58:36.962 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m5.378s 2024-03-25T19:58:36.962 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m13.923s 2024-03-25T19:58:36.962 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m0.525s 2024-03-25T19:58:37.022 INFO:teuthology.orchestra.run.smithi076.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:58:37.024 INFO:teuthology.orchestra.run.smithi076.stderr: 2024-03-25T19:58:37.024 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m5.416s 2024-03-25T19:58:37.024 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m9.307s 2024-03-25T19:58:37.025 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.302s 2024-03-25T19:58:37.025 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:58:37.025 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/remote/smithi050/log 2024-03-25T19:58:37.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:58:39.541 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/remote/smithi076/log 2024-03-25T19:58:39.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:58:41.132 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:58:41.353 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5f07cf8f48d042bf9c7b9fe53edb76e9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:58:41.359 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:58:41.367 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:58:41.382 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:58:41.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:58:41.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:58:41.484 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:58:41.493 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:58:41.493 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:58:41.495 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:58:41.509 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:58:41.509 INFO:teuthology.orchestra.run.smithi076.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:58:41.513 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:58:41.513 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2024-03-25T19:58:41.513 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 -545us[ -530us] +/- 67ms 2024-03-25T19:58:41.513 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -146us[ -131us] +/- 70ms 2024-03-25T19:58:41.513 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +756us[ +756us] +/- 73ms 2024-03-25T19:58:41.513 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:58:41.515 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:58:41.515 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-03-25T19:58:41.515 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 -489us[ -489us] +/- 68ms 2024-03-25T19:58:41.515 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -73us[ -19us] +/- 70ms 2024-03-25T19:58:41.516 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +812us[ +812us] +/- 73ms 2024-03-25T19:58:41.516 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:58:41.516 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:58:41.525 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:58:41.525 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:58:41.534 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:58:41.629 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:58:41.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:58:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:41.724 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:58:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:41.755 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:58:41.778 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:58:41.817 INFO:teuthology.task.internal:Duration was 5541.651590 seconds 2024-03-25T19:58:41.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:58:41.870 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:58:41.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:58:41.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:58:41.915 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:58:41.916 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:58:42.400 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:58:42.400 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-03-25T19:58:42.401 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:58:42.422 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2024-03-25T19:58:42.423 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:58:42.465 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:58:42.465 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:58:42.467 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:58:42.532 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:58:42.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:58:42.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:58:42.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:58:42.687 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:58:42.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:58:42.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:58:42.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:58:42.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:42.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:42.779 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-03-25T19:58:42.806 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2024-03-25T19:58:42.826 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:42.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:42.860 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:58:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:58:42.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:58:42.889 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:58:42.890 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/remote/smithi050 2024-03-25T19:58:42.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:58:42.936 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325/remote/smithi076 2024-03-25T19:58:42.936 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:58:42.971 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:58:42.971 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:58:42.974 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:58:43.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:58:43.033 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:58:43.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:58:43.043 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:58:43.043 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:58:43.046 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:58:43.057 INFO:teuthology.orchestra.run.smithi050.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:58 /home/ubuntu/cephtest 2024-03-25T19:58:43.074 INFO:teuthology.orchestra.run.smithi076.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:58 /home/ubuntu/cephtest 2024-03-25T19:58:43.075 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:58:43.086 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:58:43.108 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:58:43.135 DEBUG:teuthology.nuke:shortname: smithi050 2024-03-25T19:58:43.136 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:58:43.160 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.651068', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA9CW1XN++EKPZEb9HmxbLYSDkky3NYmzMESg0BgqpifYbRz3QKNUxTHlGiSxUPIXQ8WKSkuXfLz3ENkao/ZWKU='} 2024-03-25T19:58:43.164 DEBUG:teuthology.nuke:shortname: smithi076 2024-03-25T19:58:43.164 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:58:43.188 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621325', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:17:11.649769', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD61+DYpDp+6uYpwZwXyqjynu8sPDA81HZl0StzJkQ3mELz3Zi7fNYKjpR9ksHYe8nHoeAss37muvbvByDQjCbg='} 2024-03-25T19:58:43.213 INFO:teuthology.orchestra.console.smithi050:Power off 2024-03-25T19:58:43.213 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:58:43.239 DEBUG:teuthology.orchestra.console.smithi050:power off output: Chassis Power Control: Down/Off 2024-03-25T19:58:43.239 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:43.259 INFO:teuthology.orchestra.console.smithi076:Power off 2024-03-25T19:58:43.260 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:58:43.269 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-03-25T19:58:43.282 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2024-03-25T19:58:43.282 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:43.308 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-25T19:58:47.270 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:47.310 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:47.381 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is on 2024-03-25T19:58:47.421 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-25T19:58:51.386 DEBUG:teuthology.orchestra.console.smithi050:pexpect command: ipmitool -H smithi050.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:51.422 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:51.498 DEBUG:teuthology.orchestra.console.smithi050:check power output: Chassis Power is off 2024-03-25T19:58:51.533 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2024-03-25T19:58:51.600 INFO:teuthology.orchestra.console.smithi050:Power off completed 2024-03-25T19:58:55.534 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:58:55.645 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2024-03-25T19:58:55.746 INFO:teuthology.orchestra.console.smithi076:Power off completed 2024-03-25T19:58:55.851 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 5541.651589632034 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5f07cf8f48d042bf9c7b9fe53edb76e9 status: fail success: false 2024-03-25T19:58:55.851 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:58:55.932 INFO:teuthology.run:FAIL